Information Processing System, Game System, Information Processing Device, Terminal Device, Information Processing Method, and Storage Medium Having Information Processing Program Stored Therein

ABSTRACT

An example game system includes a game apparatus and an operating device. The game apparatus acquires operation data from the operating device. Moreover, the game apparatus executes an application and outputs an image generated by the execution of the application to either the operating device or a predetermined display device different from the operating device, or both. When a predetermined operation is performed on the operating device during the execution of the application, a predetermined operation image is displayed on a display unit of the operating device. In a state where the operation image is displayed on the display unit, the game apparatus determines whether or not a control instruction has been provided to control the predetermined display device, on the basis of the operation data. When the control instruction is determined to have been provided, the predetermined display device is controlled in accordance with the control instruction.

CROSS REFERENCE TO RELATED APPLICATION

This is a continuation of U.S. patent application Ser. No. 13/418,540filed on Mar. 13, 2012, the disclosure of which is herein incorporatedby reference in its entirety. The disclosure of Japanese PatentApplication Nos. 2011-125738 and 2011-125739, filed Jun. 3, 2011, isincorporated herein by reference.

FIELD

Disclosed herein are game systems, information processing methods, gameapparatuses, and storage media having information processing programsstored therein, which are intended to use display devices such astelevisions.

BACKGROUND AND SUMMARY

Conventionally, some game systems use televisions as display devices,and conceivably execute various applications in addition to games. Forexample, in such a conventional game system, a game apparatus mightexecute, in addition to a game program, an application for displaying atelevision program guide on the television. In this manner, game systemscan be intended for game use and also for various other uses.

Here, the television, which is also used as a display device in the gamesystem, is intended for purposes of displaying images from the gamesystem (e.g., game images and images for the program guide), and alsofor purposes other than in the game system (e.g., displaying televisionbroadcast video). Moreover, in the case where the game system can beused for various purposes as described above, the frequency of use ofthe game system itself increases, and therefore, it is quite conceivablethat the user uses the game system (e.g., to view the program guide)while watching the television. However, the conventional art asmentioned above has a problem with user-friendliness in switchingbetween a state where the television is used in the game system andanother state where the television is used for a purpose other than inthe game system.

Therefore, the game systems, information processing methods, gameapparatuses, and storage media having information processing programsstored therein as disclosed herein allow a display device, such as atelevision, to be readily operated during the execution of anapplication.

(1) An example game system described herein includes a game apparatusand an operating device.

The operating device includes an operation unit, a reception unit, and adisplay unit. The operation unit outputs operation data representing anoperation on the operating device. The reception unit receives an imageoutputted by the game apparatus. The display unit displays the receivedimage.

The game apparatus includes an acquisition unit, an applicationexecution unit, an operation image generation unit, an instructiondetermination unit, and a device control unit. The acquisition unitacquires the operation data. The application execution unit executes anapplication and outputs an image generated by the execution of theapplication to either the operating device or a predetermined displaydevice different from the operating device, or both. When apredetermined operation is performed on the operating device during theexecution of the application, the operation image generation unitgenerates and outputs a predetermined operation image to the operatingdevice. The instruction determination unit makes a determination in astate where the operation image is displayed on the display unit, on thebasis of the operation data, the determination being as to whether ornot a control instruction has been provided to control the predetermineddisplay device. When the control instruction is determined to have beenprovided, the device control unit controls the predetermined displaydevice in accordance with the control instruction.

According to the above configuration (1), when a predetermined operationis performed on an operating device during the execution of anapplication, an operation image is displayed on the screen of theoperating device. Thereafter, in the state where the operation image isdisplayed, when an instruction to control a predetermined display deviceis provided, the predetermined display device is controlled inaccordance with the control instruction. Thus, the above configuration(1) allows the predetermined display device to be operated while usingan application in the game system, without terminating the execution ofthe application. Moreover, the user can both perform the predeterminedoperation and provide the control instruction simply by using oneoperating device, so that the display device can be readily operatedeven during the execution of the application. For example, by using theoperating device, the user can readily perform operations for watching atelevision program during the execution of the application, andthereafter, resuming the application.

Note that the game apparatus may include a first image generation unitfor generating and outputting an application selection image to theoperating device. The application selection image includes imagesrepresenting applications executable by the game apparatus. In thiscase, when an application to be executed is selected with theapplication selection image being displayed, the application executionunit executes the selected application.

(2) In the state where the operation image is displayed on the displayunit, the instruction determination unit may determine whether or not atermination instruction has been provided to terminate the applicationbeing executed, on the basis of the operation data. In this case, whenthe termination instruction is determined to have been provided, theapplication execution unit terminates the execution of the application.

According to the above configuration (2), when the terminationinstruction is provided with the operation image being displayed, anapplication being executed is terminated. Thus, when the operation imageis displayed, the user can both operate the display device and performan operation to terminate the application.

(3) When a predetermined operation is performed on the operating deviceduring the execution of the application, the application execution unitmay suspend the execution of the application. In this case, with theoperation image being displayed on the display unit, the instructiondetermination unit determines whether or not a resumption instructionhas been provided to resume the execution of the application, on thebasis of the operation data. When the resumption instruction isdetermined to have been provided, the application execution unit resumesthe execution of the application.

According to the above configuration (3), when the resumptioninstruction is provided with the operation image being displayed, theexecution of an application is resumed. Accordingly, with the operationimage being displayed, the user can operate the display device, andthereafter, readily resume the application. Thus, for example, by usingthe operating device, the user can readily perform a series ofoperations of watching the television while using the application, andthereafter, resuming the use of the application.

(4) The predetermined display device may be capable of receivingtelevision broadcasting and displaying a television program. In thiscase, when the operation image is displayed on the display unit, theoperation image generation unit generates an image including a programguide and outputs the generated image to the predetermined displaydevice.

According to the above configuration (4), when the operation image isdisplayed on the operating device, a television program guide isdisplayed on the predetermined display device. Thus, the user canoperate the television (the predetermined display device) while viewingthe program guide, which facilitates the operation of, for example,selecting the channel of the television.

(5) The operation image generation unit may generate as the operationimage an image including a control instruction image representing thecontrol instruction. In this case, when the control instruction image isselected, the instruction determination unit determines the controlinstruction to have been provided.

According to the above configuration (5), the operating device displaysa control instruction image representing the control instruction as theoperation image. Thus, the user can more readily provide the controlinstruction.

(6) The predetermined display device may be capable of receivingtelevision broadcasting and displaying a television program. In thiscase, the operation image generation unit generates an image including aprogram guide as the operation image, and outputs the generated image tothe operating device.

According to the above configuration (6), the operating device displaysa program guide as the operation image. Thus, the user can operate thetelevision (the predetermined display device) while viewing the programguide, which facilitates the operation of, for example, selecting thechannel of the television.

(7) The predetermined display device may be switchable between a firstmode where an image inputted from the game apparatus is displayed and asecond mode where a television broadcast image is displayed. In thiscase, the device control unit controls the predetermined display deviceto be switched to the second mode in response to the operating devicedisplaying the operation image.

According to the above configuration (7), when switching display to theoperation image, the user does not perform any input switching operationon the predetermined display device, resulting in more simplified useroperations.

(8) The predetermined display device may be switchable between a firstmode where an image inputted from the game apparatus is displayed and asecond mode where a television broadcast image is displayed. The devicecontrol unit controls the predetermined display device to be switched tothe first mode in response to the operation image displayed on theoperating device switching to an image generated by the execution of theapplication.

According to the above configuration (8), when switching display fromthe operation image to an image for an application, the user does notperform any input switching operation on the predetermined displaydevice, resulting in more simplified user operations.

(9) The operating device may further include a control signal outputunit for outputting a control signal to control the predetermineddisplay device. In this case, the device control unit instructs theoperating device to cause the control signal output unit to transmit acontrol signal indicating the content of control according to thecontrol instruction.

The above configuration (9) makes it possible to readily control thepredetermined display device from the game apparatus via the operatingdevice.

(10) The control signal output unit may output an infrared signal as thecontrol signal. In this case, the predetermined display device includesan infrared light reception unit capable of receiving the infraredsignal.

The above configuration (10) allows the operating device to control thepredetermined display device by the infrared signal. Note that generaltelevisions (such as a television 2 in the example embodiment to bedescribed later) have the function of receiving an infrared signal andperform an operation specified by the infrared signal. Thus, accordingto the above configuration (10), a television can be used as thepredetermined display device, so that a highly versatile game system canbe provided.

(11) The device control unit may output a control signal to thepredetermined display device, the signal indicating the content ofcontrol according to the control instruction.

The above configuration (11) allows the information processing apparatusto readily control the predetermined display device by transmitting acontrol command. According to the above configuration (11), for example,a television which supports the HDMI standard can be used as thepredetermined display device, so that a highly versatile game system canbe provided.

(12) The operating device may further include a touch panel provided ona screen of the display unit. In this case, the operation imagegeneration unit generates as the operation image an image including acontrol instruction image representing the control instruction. When thecontrol instruction image is selected on the touch panel, theinstruction determination unit determines the control instruction tohave been provided.

The above configuration (12) allows the user to provide the controlinstruction using the touch panel. Thus, the user can provideinstructions by easy and intuitive operations using the touch panel.

Note that disclosed herein is an example of the game apparatus includedin the game system as described in (1) to (12) above. Also disclosedherein is an example non-transitory storage medium having stored thereina game program (an information processing program) for causing acomputer of the game apparatus to function as means equivalent to theaforementioned units of the game apparatus (excluding the acquisitionunit and the operation image output unit). Also disclosed herein is anexample game processing method to be executed in the game system orapparatus as described in (1) to (12) above.

In the game system, information processing method, game apparatus,storage medium having an information processing program stored thereinas mentioned above, an operation image is displayed on the screen of anoperating device in response to a predetermined operation beingperformed on the operating device during the execution of anapplication, so that, with the operation image being displayed, apredetermined display device can be controlled in accordance with acontrol instruction. Thus, when using an application in the game system,the predetermined display device can be operated without terminating theexecution of the application.

These and other objects, features, aspects and advantages will becomemore apparent from the following detailed description when taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an external view of an example non-limiting game system;

FIG. 2 is a block diagram illustrating an internal configuration of anexample non-limiting game apparatus;

FIG. 3 is a perspective view illustrating an external configuration ofan example non-limiting controller;

FIG. 4 is another perspective view illustrating the externalconfiguration of the controller;

FIG. 5 is a diagram illustrating an internal configuration of thecontroller;

FIG. 6 is another diagram illustrating the internal configuration of thecontroller;

FIG. 7 is a block diagram illustrating a configuration of thecontroller;

FIG. 8 is a diagram illustrating an external configuration of an examplenon-limiting terminal device;

FIG. 9 is another diagram illustrating the external configuration of theterminal device;

FIG. 10 is a diagram illustrating an example of the terminal devicebeing held by the user in a landscape position;

FIG. 11 is a block diagram illustrating an internal configuration of theterminal device;

FIG. 12 is a block diagram illustrating an example connectionrelationship between the game system and an external device;

FIG. 13 is a state transition diagram representing transitions betweenoperation states of the game system;

FIG. 14 is a diagram illustrating an example storage area of a mainmemory in the game apparatus;

FIG. 15 is a flowchart illustrating an example flow of an initialprocess performed by the game apparatus;

FIG. 16 is a flowchart illustrating an example flow of a menu processperformed by the game apparatus;

FIG. 17 is a diagram illustrating an example app selection imagedisplayed on the terminal device;

FIG. 18 is a flowchart illustrating an example flow of a televisionprocess performed by the game apparatus 3;

FIG. 19 is a diagram illustrating an example device control imagedisplayed on the terminal device;

FIG. 20 is a flowchart illustrating an example flow of a browser processperformed by the game apparatus;

FIG. 21 is a diagram illustrating an example browser image displayed onthe terminal device;

FIG. 22 is a flowchart illustrating an example flow of an appinterruption process performed by the game apparatus;

FIG. 23 is a diagram illustrating an example setting image displayed onthe terminal device; and

FIG. 24 is a diagram illustrating an example remote control imagedisplayed on the terminal device.

DETAILED DESCRIPTION OF NON-LIMITING EXAMPLE EMBODIMENTS 1. OverallConfiguration of the Game System

An example game system 1 according to an example embodiment will now bedescribed with reference to the drawings. FIG. 1 is an external view ofthe game system 1. In FIG. 1, the game system 1 includes a stationarydisplay device (hereinafter referred to as a “television”) 2 such as atelevision receiver, a stationary game apparatus 3, an optical disc 4, acontroller 5, a marker device 6, and a terminal device 7. In the gamesystem 1, the game apparatus 3 performs game processes based on gameoperations performed using the controller 5 and the terminal device 7,and game images acquired through the game processes are displayed on thetelevision 2 and/or the terminal device 7.

In the game apparatus 3, the optical disc 4 typifying an informationstorage medium used for the game apparatus 3 in a replaceable manner isremovably inserted. An information processing program (a game program,for example) to be executed by the game apparatus 3 is stored in theoptical disc 4. The game apparatus 3 has, on the front surface thereof,an insertion opening for the optical disc 4. The game apparatus 3 readsand executes the information processing program stored on the opticaldisc 4 which is inserted into the insertion opening, to perform the gameprocess.

The television 2 is connected to the game apparatus 3 by a connectingcord. Game images acquired as a result of the game processes performedby the game apparatus 3 are displayed on the television 2. Thetelevision 2 includes speakers 2 a (see FIG. 2), and the speakers 2 aoutput game sounds acquired as a result of the game process. Inalternative example embodiments, the game apparatus 3 and the stationarydisplay device may be an integral section. Also, the communicationbetween the game apparatus 3 and the television 2 may be wirelesscommunication.

The marker device 6 is provided along the periphery of the screen (onthe upper side of the screen in FIG. 1) of the television 2. The user(player) can perform game operations by moving the controller 5, thedetails of which will be described later, and the marker device 6 isused by the game apparatus 3 for calculating the movement, position,attitude, etc., of the controller 5. The marker device 6 includes twomarkers 6R and 6L on opposite ends thereof. Specifically, the marker 6R(as well as the marker 6L) includes one or more infrared light emittingdiodes (LEDs), and emits an infrared light in a forward direction fromthe television 2. The marker device 6 is connected to the game apparatus3 by wire (or wirelessly), and the game apparatus 3 is able to controlthe lighting of each infrared LED of the marker device 6. Note that themarker device 6 is of a portable type so that the user can install themarker device 6 in any desired position. While FIG. 1 shows an exampleembodiment in which the marker device 6 is arranged on top of thetelevision 2, the position and the direction of arranging the markerdevice 6 are not limited to this particular arrangement.

The controller 5 provides the game apparatus 3 with operation datarepresenting the content of operations performed on the controlleritself. The controller 5 and the game apparatus 3 can wirelesslycommunicate with each other. In the present example embodiment, thewireless communication between the controller 5 and the game apparatus 3uses, for example, Bluetooth (Registered Trademark) technology. In otherexample embodiments, the controller 5 and the game apparatus 3 may beconnected by a wired connection. Furthermore, in FIG. 1, the game system1 includes only one controller 5, but the game system 1 may include morethan one controller 5. Specifically, the game apparatus 3 is capable ofcommunicating with a plurality of controllers, so that by using apredetermined number of controllers at the same time, a plurality ofpeople can play the game. The configuration of the controller 5 will bedescribed in detail later.

The terminal device 7 is a portable device of a size that can be held bythe user, so that the user can hold and move the terminal device 7 orcan place the terminal device 7 in any desired position. As will bedescribed in detail later, the terminal device 7 includes a liquidcrystal display (LCD) 51, and input means (e.g., a touch panel 52 and agyroscope 74 to be described later). The terminal device 7 cancommunicate with the game apparatus 3 wirelessly (or wired). Theterminal device 7 receives data for images generated by the gameapparatus 3 (e.g., game images) from the game apparatus 3, and displaysthe images on the LCD 51. Note that in the present example embodiment,the LCD is used as the display of the terminal device 7, but theterminal device 7 may include any other display device, e.g., a displaydevice utilizing electro luminescence (EL). Furthermore, the terminaldevice 7 transmits operation data representing the content of operationsperformed thereon to the game apparatus 3.

2. Internal Configuration of the Game Apparatus 3

An internal configuration of the game apparatus 3 will be described withreference to FIG. 2. FIG. 2 is a block diagram illustrating an internalconfiguration of the game apparatus 3. The game apparatus 3 includes aCPU (Central Processing Unit) 10, a system LSI 11, external main memory12, a ROM/RTC 13, a disc drive 14, and an AV-IC 15.

The CPU 10 performs game processes by executing a game program stored,for example, on the optical disc 4, and functions as a game processor.The CPU 10 is connected to the system LSI 11. The external main memory12, the ROM/RTC 13, the disc drive 14, and the AV-IC 15, as well as theCPU 10, are connected to the system LSI 11. The system LSI 11 performsprocesses for controlling data transmission between the respectivecomponents connected thereto, generating images to be displayed,acquiring data from an external device(s), and the like. The internalconfiguration of the system LSI 11 will be described below. The externalmain memory 12 is of a volatile type and stores a program such as a gameprogram read from the optical disc 4, a game program read from flashmemory 17, and various data. The external main memory 12 is used as awork area and a buffer area for the CPU 10. The ROM/RTC 13 includes aROM (a so-called boot ROM) incorporating a boot program for the gameapparatus 3, and a clock circuit (RTC: Real Time Clock) for countingtime. The disc drive 14 reads program data, texture data, and the likefrom the optical disc 4, and writes the read data into internal mainmemory 11 e (to be described below) or the external main memory 12.

The system LSI 11 includes an input/output processor (I/O processor) 11a, a GPU (Graphics Processor Unit) 11 b, a DSP (Digital SignalProcessor) 11 c, VRAM (Video RAM) 11 d, and the internal main memory 11e. Although not shown in the figures, these components 11 a to 11 e areconnected with each other through an internal bus.

The GPU 11 b, acting as a part of a rendering mechanism, generatesimages in accordance with graphics commands (rendering commands) fromthe CPU 10. The VRAM 11 d stores data (data such as polygon data andtexture data) to be used by the GPU 11 b to execute the graphicscommands. When images are generated, the GPU 11 b generates image datausing data stored in the VRAM 11 d. Note that in the present exampleembodiment, the game apparatus 3 generates both images (game images) tobe displayed on the television 2 and images (game images) to bedisplayed on the terminal device 7. Hereinafter, the images to bedisplayed on the television 2 are referred to as the “television images”and the images to be displayed on the terminal device 7 are referred toas the “terminal images”.

The DSP 11 c, functioning as an audio processor, generates sound datausing sound data and sound waveform (e.g., tone quality) data stored inone or both of the internal main memory 11 e and the external mainmemory 12. Note that in the present example embodiment, sounds (gamesounds) to be generated are classified into two types as in the case ofthe game images, one being outputted from the speaker of the television2, the other being outputted from speakers of the terminal device 7.Hereinafter, in some cases, the sounds to be outputted from thetelevision 2 are referred to as “television sounds”, and the sounds tobe outputted from the terminal device 7 are referred to as “terminalsounds”.

Among the images and sounds generated by the game apparatus 3 asdescribed above, both image data and sound data to be outputted from thetelevision 2 are read out by the AV-IC 15. The AV-IC 15 outputs theread-out image data to the television 2 via an AV connector 16, andoutputs the read-out sound data to the speakers 2 a provided in thetelevision 2. Thus, images are displayed on the television 2, and soundsare outputted from the speakers 2 a. Note that the game apparatus 3 andthe television 2 may be connected in any manner, and a control commandfor controlling the television 2 may be transmitted to the television 2by the game apparatus 3 in a wired or wireless manner. For example, anHDMI cable, which supports the HDMI (high-definition multimediainterface) standard, may be used. The HDMI standard allows a device tocontrol another device connected thereto on the basis of a functioncalled CEC (consumer electronics control). Accordingly, in the casewhere the HDMI cable is used so that the game apparatus 3 can controlthe television 2, the game apparatus 3 can turn on the television 2 orswitch between inputs to the television 2 at appropriate times.

Furthermore, among the images and sounds generated by the game apparatus3, both image data and sound data to be outputted by the terminal device7 are transmitted to the terminal device 7 by the input/output processor11 a, etc. The data transmission to the terminal device 7 by theinput/output processor 11 a, etc., will be described later.

The input/output processor 11 a exchanges data with components connectedthereto, and downloads data from an external device(s). The input/outputprocessor 11 a is connected to the flash memory 17, a networkcommunication module 18, a controller communication module 19, anexpansion connector 20, a memory card connector 21, and a codec LSI 27.Furthermore, an antenna 22 is connected to the network communicationmodule 18. An antenna 23 is connected to the controller communicationmodule 19. The codec LSI 27 is connected to a terminal communicationmodule 28, and an antenna 29 is connected to the terminal communicationmodule 28.

The game apparatus 3 is capable of connecting to a network such as theInternet to communicate with an external information processingapparatus (e.g., another game apparatus or a server). Specifically, theinput/output processor 11 a can be connected to a network such as theInternet via the network communication module 18 and the antenna 22 tocommunicate with other information processing apparatuses connected tothe network. The input/output processor 11 a regularly accesses theflash memory 17, and detects the presence or absence of any data to betransmitted to the network, and when detected, transmits the data to thenetwork via the network communication module 18 and the antenna 22.Further, the input/output processor 11 a receives data transmitted fromthe external information processing apparatuses and data downloaded froma download server via the network, the antenna 22 and the networkcommunication module 18, and stores the received data in the flashmemory 17. The CPU 10 executes a game program so as to read data storedin the flash memory 17 and use the data, as appropriate, in the gameprogram. The flash memory 17 may store game save data (e.g., game resultdata or unfinished game data) of a game played using the game apparatus3 in addition to data exchanged between the game apparatus 3 and theexternal information processing apparatuses. Moreover, the flash memory17 may have a game program stored therein.

Furthermore, the game apparatus 3 is capable of receiving operation datafrom the controller 5. Specifically, the input/output processor 11 areceives operation data transmitted from the controller 5 via theantenna 23 and the controller communication module 19, and stores it(temporarily) in a buffer area of the internal main memory 11 e or theexternal main memory 12.

Furthermore, the game apparatus 3 is capable of exchanging data, forimages, sound, etc., with the terminal device 7. When transmitting gameimages (terminal game images) to the terminal device 7, the input/outputprocessor 11 a outputs game image data generated by the GPU 11 b to thecodec LSI 27. The codec LSI 27 performs a predetermined compressionprocess on the image data from the input/output processor 11 a. Theterminal communication module 28 wirelessly communicates with theterminal device 7. Accordingly, the image data compressed by the codecLSI 27 is transmitted by the terminal communication module 28 to theterminal device 7 via the antenna 29. In the present example embodiment,the image data transmitted from the game apparatus 3 to the terminaldevice 7 is image data used in a game, and the playability of a game canbe adversely influenced if there is a delay in the images displayed inthe game. Therefore, the game system avoids delay in transmitting imagedata from the game apparatus 3 to the terminal device 7. Therefore, inthe present example embodiment, the codec LSI 27 compresses image datausing a compression technique with high efficiency such as the H.264standard, for example. Other compression techniques may be used, andimage data may be transmitted uncompressed if the communication speed issufficient. The terminal communication module 28 is, for example, aWi-Fi certified communication module, and may perform wirelesscommunication at high speed with the terminal device 7 using a MIMO(Multiple Input Multiple Output) technique employed in the IEEE 802.11nstandard, for example, or may use other communication schemes.

Furthermore, in addition to the image data, the game apparatus 3 alsotransmits sound data to the terminal device 7. Specifically, theinput/output processor 11 a outputs sound data generated by the DSP 11 cto the terminal communication module 28 via the codec LSI 27. The codecLSI 27 performs a compression process on the sound data as it does onthe image data. Any method can be employed for compressing the sounddata, and such a method uses a high compression rate but causes lesssound degradation. Also, in another example embodiment, the sound datamay be transmitted without compression. The terminal communicationmodule 28 transmits compressed image and sound data to the terminaldevice 7 via the antenna 29.

Furthermore, in addition to the image and sound data, the game apparatus3 transmits various control data to the terminal device 7 whereappropriate. The control data is data representing an instruction tocontrol a component included in the terminal device 7, e.g., aninstruction to control lighting of a marker section (a marker section 55shown in FIG. 11) or an instruction to control shooting by a camera (acamera 56 shown in FIG. 11). The input/output processor 11 a transmitsthe control data to the terminal device 7 in accordance with aninstruction from the CPU 10. Note that in the present exampleembodiment, the codec LSI 27 does not perform a compression process onthe control data, but in another example embodiment, a compressionprocess may be performed. Note that the data to be transmitted from thegame apparatus 3 to the terminal device 7 may or may not be codeddepending on the situation.

Furthermore, the game apparatus 3 is capable of receiving various datafrom the terminal device 7. As will be described in detail later, in thepresent example embodiment, the terminal device 7 transmits operationdata, image data, and sound data. The data transmitted by the terminaldevice 7 is received by the terminal communication module 28 via theantenna 29. Here, the image data and the sound data from the terminaldevice 7 have been subjected to the same compression process asperformed on the image data and the sound data from the game apparatus 3to the terminal device 7. Accordingly, the image data and the sound dataare transferred from the terminal communication module 28 to the codecLSI 27, and subjected to a decompression process by the codec LSI 27before output to the input/output processor 11 a. On the other hand, theoperation data from the terminal device 7 is smaller in size than theimage data or the sound data and therefore is not always subjected to acompression process. Moreover, the operation data may or may not becoded depending on the situation. Accordingly, after being received bythe terminal communication module 28, the operation data is outputted tothe input/output processor 11 a via the codec LSI 27. The input/outputprocessor 11 a stores the data received from the terminal device 7(temporarily) in a buffer area of the internal main memory 11 e or theexternal main memory 12.

Furthermore, the game apparatus 3 can be connected to other devices orexternal storage media. Specifically, the input/output processor 11 a isconnected to the expansion connector 20 and the memory card connector21. The expansion connector 20 is a connector for an interface, such asa USB or SCSI interface. The expansion connector 20 can receive a mediumsuch as an external storage medium, a peripheral device such as anothercontroller, or a wired communication connector which enablescommunication with a network in place of the network communicationmodule 18. The memory card connector 21 is a connector for connectingthereto an external storage medium such as a memory card (which may beof a proprietary or standard format, such as SD, miniSD, microSD,Compact Flash, etc.). For example, the input/output processor 11 a canaccess an external storage medium via the expansion connector 20 or thememory card connector 21 to store data in the external storage medium orread data from the external storage medium.

The game apparatus 3 includes a power button 24, a reset button 25, andan eject button 26. The power button 24 and the reset button 25 areconnected to the system LSI 11. When the power button 24 is on, power issupplied from an external power source to the components of the gameapparatus 3 via an AC adaptor (not shown). When the reset button 25 ispressed, the system LSI 11 restarts a boot program of the game apparatus3. The eject button 26 is connected to the disc drive 14. When the ejectbutton 26 is pressed, the optical disc 4 is ejected from the disc drive14.

In other example embodiments, some of the components of the gameapparatus 3 may be provided as extension devices separate from the gameapparatus 3. In this case, an extension device may be connected to thegame apparatus 3 via the expansion connector 20, for example.Specifically, an extension device may include components as describedabove, e.g., a codec LSI 27, a terminal communication module 28, and anantenna 29, and can be attached to/detached from the expansion connector20. Thus, by connecting the extension device to a game apparatus whichdoes not include the above components, the game apparatus cancommunicate with the terminal device 7.

3. Configuration of the Controller 5

Next, with reference to FIGS. 3 to 7, the controller 5 will bedescribed. FIGS. 3 and 4 are perspective views each illustrating anexternal configuration of the controller 5. The perspective view of FIG.3 shows the controller 5 as viewed from the top rear side thereof, andthe perspective view of FIG. 4 shows the controller 5 as viewed from thebottom front side thereof.

As shown in FIG. 3 and FIG. 4, the controller 5 has a housing 31 formedby, for example, plastic molding. The housing 31 has a generallyparallelepiped shape extending in a longitudinal direction from front torear (Z-axis direction shown in FIG. 3), and as a whole is sized to beheld by one hand of an adult or even a child. The user can perform gameoperations by pressing buttons provided on the controller 5, and movingthe controller 5 to change the position and the attitude (tilt) thereof.

The housing 31 has a plurality of operation buttons. As shown in FIG. 3,on the top surface of the housing 31, a cross button 32 a, a firstbutton 32 b, a second button 32 c, an A button 32 d, a minus button 32e, a home button 32 f, a plus button 32 g, and a power button 32 h areprovided. In the present example embodiment, the top surface of thehousing 31 on which the buttons 32 a to 32 h are provided may bereferred to as a “button surface”. On the other hand, as shown in FIG.4, a recessed portion is formed on the bottom surface of the housing 31,and a B button 32 i is provided on a rear slope surface of the recessedportion. The operation buttons 32 a to 32 i are appropriately assignedtheir respective functions in accordance with the information processingprogram executed by the game apparatus 3. Further, the power button 32 his intended to remotely turn ON/OFF the game apparatus 3. The homebutton 32 f and the power button 32 h each have the top surface thereofrecessed below the top surface of the housing 31. Therefore, the homebutton 32 f and the power button 32 h are prevented from beinginadvertently pressed by the user.

On the rear surface of the housing 31, the connector 33 is provided. Theconnector 33 is used for connecting the controller 5 to another device(e.g., another sensor section or controller). Both sides of theconnector 33 on the rear surface of the housing 31 have a fastening hole33 a for preventing easy inadvertent disengagement of another device asdescribed above.

In the rear-side portion of the top surface of the housing 31, aplurality (four in FIG. 3) of LEDs 34 a, 34 b, 34 c, and 34 d areprovided. The controller 5 is assigned a controller type (number) so asto be distinguishable from another controller. The LEDs 34 a, 34 b, 34c, and 34 d are each used for informing the user of the controller typewhich is currently being set for the controller 5 being used, and forinforming the user of remaining battery power of the controller 5, forexample. Specifically, when a game operation is performed using thecontroller 5, one of the LEDs 34 a, 34 b, 34 c, and 34 d correspondingto the controller type is lit up.

The controller 5 has an imaging information calculation section 35 (FIG.6), and a light incident surface 35 a through which a light is incidenton the imaging information calculation section 35 is provided on thefront surface of the housing 31, as shown in FIG. 4. The light incidentsurface 35 a is made of a material transmitting therethrough at leastinfrared light outputted from the markers 6R and 6L.

On the top surface of the housing 31, sound holes 31 a for externallyoutputting a sound from a speaker 47 (shown in FIG. 5) incorporated inthe controller 5 is provided between the first button 32 b and the homebutton 32 f.

Next, with reference to FIGS. 5 and 6, an internal configuration of thecontroller 5 will be described. FIG. 5 and FIG. 6 are diagramsillustrating the internal configuration of the controller 5. FIG. 5 is aperspective view illustrating a state where an upper casing (a part ofthe housing 31) of the controller 5 is removed. FIG. 6 is a perspectiveview illustrating a state where a lower casing (a part of the housing31) of the controller 5 is removed. The perspective view of FIG. 6 showsa substrate 30 of FIG. 5 as viewed from the reverse side.

As shown in FIG. 5, the substrate 30 is fixed inside the housing 31, andon a top main surface of the substrate 30, the operation buttons 32 a to32 h, the LEDs 34 a, 34 b, 34 c, and 34 d, an acceleration sensor 37, anantenna 45, the speaker 47, and the like are provided. These elementsare connected to a microcomputer 42 (see FIG. 6) via lines (not shown)formed on the substrate 30 and the like. In the present exampleembodiment, the acceleration sensor 37 is provided on a position offsetfrom the center of the controller 5 with respect to the X-axisdirection. Thus, calculation of the motion of the controller 5 beingrotated about the Z-axis may be facilitated. Further, the accelerationsensor 37 is provided anterior to the center of the controller 5 withrespect to the longitudinal direction (Z-axis direction). Further, awireless module 44 (see FIG. 6) and the antenna 45 allow the controller5 to act as a wireless controller.

On the other hand, as shown in FIG. 6, at a front edge of a bottom mainsurface of the substrate 30, the imaging information calculation section35 is provided. The imaging information calculation section 35 includesan infrared filter 38, a lens 39, an image pickup element 40 and animage processing circuit 41 located in order, respectively, from thefront of the controller 5. These components 38 to 41 are attached on thebottom main surface of the substrate 30.

On the bottom main surface of the substrate 30, the microcomputer 42 anda vibrator 46 are provided. The vibrator 46 is, for example, a vibrationmotor or a solenoid, and is connected to the microcomputer 42 via linesformed on the substrate 30 or the like. The controller 5 is vibrated byactuation of the vibrator 46 based on a command from the microcomputer42. Therefore, the vibration is conveyed to the user's hand holding thecontroller 5, and thus a so-called vibration-feedback game is realized.In the present example embodiment, the vibrator 46 is disposed slightlytoward the front of the housing 31. That is, the vibrator 46 ispositioned offset from the center toward the end of the controller 5,and therefore the vibration of the vibrator 46 can lead to enhancementof the vibration of the entire controller 5. Further, the connector 33is provided at the rear edge of the bottom main surface of the substrate30. In addition to the components shown in FIGS. 5 and 6, the controller5 includes a quartz oscillator for generating a reference clock of themicrocomputer 42, an amplifier for outputting a sound signal to thespeaker 47, and the like.

FIGS. 3 to 6 only show examples of the shape of the controller 5, theshape of each operation button, the number and the positions ofacceleration sensors and vibrators, and so on, and other shapes,numbers, and positions may be employed. Further, although in the presentexample embodiment the imaging direction of the image pickup means isthe Z-axis positive direction, the imaging direction may be anydirection. That is, the imagining information calculation section 35(the light incident surface 35 a through which a light is incident onthe imaging information calculation section 35) of the controller 5 maynot necessarily be provided on the front surface of the housing 31, butmay be provided on any other surface on which a light can be receivedfrom the outside of the housing 31.

FIG. 7 is a block diagram illustrating a configuration of the controller5. The controller 5 includes an operating section 32 (the operationbuttons 32 a to 32 i), the imaging information calculation section 35, acommunication section 36, the acceleration sensor 37, and a gyroscope48. The controller 5 transmits, as operation data, data representing thecontent of an operation performed on the controller 5 itself, to thegame apparatus 3. Note that hereinafter, in some cases, operation datatransmitted by the controller 5 is referred to as “controller operationdata”, and operation data transmitted by the terminal device 7 isreferred to as “terminal operation data”.

The operating section 32 includes the operation buttons 32 a to 32 idescribed above, and outputs, to the microcomputer 42 of thecommunication section 36, operation button data indicating an inputstate (that is, whether or not each operation button 32 a to 32 i ispressed) of each operation button 32 a to 32 i.

The imaging information calculation section 35 is a system for analyzingimage data taken by the image pickup means and calculating, for example,the centroid and the size of an area having a high brightness in theimage data. The imaging information calculation section 35 has a maximumsampling period of, for example, about 200 frames/sec., and thereforecan trace and analyze even a relatively fast motion of the controller 5.

The imaging information calculation section 35 includes the infraredfilter 38, the lens 39, the image pickup element 40 and the imageprocessing circuit 41. The infrared filter 38 transmits therethroughonly infrared light included in the light incident on the front surfaceof the controller 5. The lens 39 collects the infrared light transmittedthrough the infrared filter 38 so as to be incident on the image pickupelement 40. The image pickup element 40 is a solid-state imaging devicesuch as, for example, a CMOS sensor or a CCD sensor, which receives theinfrared light collected by the lens 39, and outputs an image signal.The marker section 55 of the terminal device 7 and the marker device 6,which are subjects to be imaged, include markers for outputting infraredlight. Therefore, the infrared filter 38 enables the image pickupelement 40 to receive only the infrared light transmitted through theinfrared filter 38 and generate image data, so that an image of eachsubject to be imaged (the marker section 55 and/or the marker device 6)can be taken with enhanced accuracy. Hereinafter, the image taken by theimage pickup element 40 is referred to as a pickup image. The image datagenerated by the image pickup element 40 is processed by the imageprocessing circuit 41. The image processing circuit 41 calculates, inthe pickup image, the positions of subjects to be imaged. The imageprocessing circuit 41 outputs data representing coordinate points of thecalculated positions, to the microcomputer 42 of the communicationsection 36. The data representing the coordinate points is transmittedas operation data to the game apparatus 3 by the microcomputer 42.Hereinafter, the coordinate points are referred to as “marker coordinatepoints”. The marker coordinate point changes depending on the attitude(angle of tilt) and/or the position of the controller 5 itself, andtherefore the game apparatus 3 is allowed to calculate the attitude andthe position of the controller 5 using the marker coordinate point.

In another example embodiment, the controller 5 may not necessarilyinclude the image processing circuit 41, and the controller 5 maytransmit the pickup image as it is to the game apparatus 3. At thistime, the game apparatus 3 may have a circuit or a program, having thesame function as the image processing circuit 41, for calculating themarker coordinate point.

The acceleration sensor 37 detects accelerations (including agravitational acceleration) of the controller 5, that is, force(including gravity) applied to the controller 5. The acceleration sensor37 detects a value of an acceleration (linear acceleration) applied to adetection section of the acceleration sensor 37 in the straight linedirection along the sensing axis direction, among all accelerationsapplied to a detection section of the acceleration sensor 37. Forexample, a multiaxial acceleration sensor having two or more axesdetects an acceleration of a component for each axis, as theacceleration applied to the detection section of the accelerationsensor. The acceleration sensor 37 is, for example, a capacitive MEMS(Micro-Electro Mechanical System) acceleration sensor. However, anothertype of acceleration sensor may be used.

In the present example embodiment, the acceleration sensor 37 detects alinear acceleration in each of three axis directions, i.e., the up/downdirection (Y-axis direction shown in FIG. 3), the left/right direction(the X-axis direction shown in FIG. 3), and the forward/backwarddirection (the Z-axis direction shown in FIG. 3), relative to thecontroller 5. The acceleration sensor 37 detects acceleration in thestraight line direction along each axis, and an output from theacceleration sensor 37 represents a value of the linear acceleration foreach of the three axes. In other words, the detected acceleration isrepresented as a three-dimensional vector in an XYZ-coordinate system(controller coordinate system) defined relative to the controller 5.

Data (acceleration data) representing the acceleration detected by theacceleration sensor 37 is outputted to the communication section 36. Theacceleration detected by the acceleration sensor 37 changes depending onthe attitude (angle of tilt) and movement of the controller 5, andtherefore the game apparatus 3 is allowed to calculate the attitude andmovement of the controller 5 using the acquired acceleration data. Inthe present example embodiment, the game apparatus 3 calculates theattitude, angle of tilt, etc., of the controller 5 based on the acquiredacceleration data.

When a computer such as a processor (e.g., the CPU 10) of the gameapparatus 3 or a processor (e.g., the microcomputer 42) of thecontroller 5 processes an acceleration signal outputted from theacceleration sensor 37 (or similarly from an acceleration sensor 73 tobe described later), additional information relating to the controller 5can be inferred or calculated (determined), as one skilled in the artwill readily understand from the description herein. For example, in thecase where the computer performs processing on the premise that thecontroller 5 including the acceleration sensor 37 is in static state(that is, in the case where processing is performed on the premise thatthe acceleration to be detected by the acceleration sensor includes onlythe gravitational acceleration), when the controller 5 is actually instatic state, it is possible to determine whether or not, or how muchthe controller 5 tilts relative to the direction of gravity, based onthe acceleration having been detected. Specifically, when the statewhere the detection axis of the acceleration sensor 37 faces verticallydownward is set as a reference, whether or not the controller 5 tiltsrelative to the reference can be determined based on whether or not 1G(gravitational acceleration) is applied to the detection axis, and thedegree to which the controller 5 tilts relative to the reference can bedetermined based on the magnitude of the gravitational acceleration.Further, the multiaxial acceleration sensor 37 processes theacceleration signals having been detected for the respective axes so asto more specifically determine the degree to which the controller 5tilts relative to the direction of gravity. In this case, the processormay calculate, based on the output from the acceleration sensor 37, theangle at which the controller 5 tilts, or the direction in which thecontroller 5 tilts without calculating the angle of tilt. Thus, theacceleration sensor is used in combination with the processor, making itpossible to determine the angle of tilt or the attitude of thecontroller 5.

On the other hand, when it is premised that the controller 5 is indynamic state (where the controller 5 is being moved), the accelerationsensor 37 detects the acceleration based on movement of the controller5, in addition to the gravitational acceleration. Therefore, when thegravitational acceleration component is eliminated from the detectedacceleration through a predetermined process, it is possible todetermine the direction in which the controller 5 moves. Even when it ispremised that the controller 5 is in dynamic state, the accelerationcomponent based on movement of the acceleration sensor is eliminatedfrom the detected acceleration through a predetermined process, wherebyit is possible to determine the tilt of the controller 5 relative to thedirection of gravity. In another example embodiment, the accelerationsensor 37 may include an embedded processor or another type of dedicatedprocessor for performing any desired processing on an accelerationsignal detected by the acceleration detection means incorporated thereinbefore outputting to the microcomputer 42. For example, when theacceleration sensor 37 is intended to detect static acceleration (e.g.,gravitational acceleration), the embedded or dedicated processor couldconvert the acceleration signal to a corresponding angle of tilt (oranother appropriate parameter).

The gyroscope 48 detects angular rates about three axes (in the presentexample embodiment, the X-, Y-, and Z-axes). In the presentspecification, the directions of rotation about the X-axis, the Y-axis,and the Z-axis relative to the imaging direction (the Z-axis positivedirection) of the controller 5 are referred to as a pitch direction, ayaw direction, and a roll direction, respectively. So long as thegyroscope 48 can detect the angular rates about the three axes, anynumber thereof may be used, and also any combination of sensors may beincluded therein. That is, the two-axis gyroscope 55 detects angularrates in the pitch direction (the direction of rotation about theX-axis) and the roll direction (the direction of rotation about theZ-axis), and the one-axis gyroscope 56 detects an angular rate in theyaw direction (the direction of rotation about the Y-axis). For example,the gyroscope 48 may be a three-axis gyroscope or may include acombination of a two-axis gyroscope and a one-axis gyroscope to detectthe angular rates about the three axes. Data representing the angularrates detected by the gyroscope 48 is outputted to the communicationsection 36. Alternatively, the gyroscope 48 may simply detect an angularrate about one axis or angular rates about two axes.

The communication section 36 includes the microcomputer 42, memory 43,the wireless module 44 and the antenna 45. The microcomputer 42 controlsthe wireless module 44 for wirelessly transmitting, to the gameapparatus 3, data acquired by the microcomputer 42 while using thememory 43 as a storage area in the process.

Data outputted from the operating section 32, the imaging informationcalculation section 35, the acceleration sensor 37, and the gyroscope 48to the microcomputer 42 is temporarily stored to the memory 43. The datais transmitted as operation data (controller operation data) to the gameapparatus 3. Specifically, at the time of the transmission to thecontroller communication module 19 of the game apparatus 3, themicrocomputer 42 outputs the operation data stored in the memory 43 tothe wireless module 44. The wireless module 44 uses, for example, theBluetooth (registered trademark) technology to modulate the operationdata onto a carrier wave of a predetermined frequency, and radiates thelow power radio wave signal from the antenna 45. That is, the operationdata is modulated onto the low power radio wave signal by the wirelessmodule 44 and transmitted from the controller 5. The controllercommunication module 19 of the game apparatus 3 receives the low powerradio wave signal. The game apparatus 3 demodulates or decodes thereceived low power radio wave signal to acquire the operation data. TheCPU 10 of the game apparatus 3 performs the game process using theoperation data acquired from the controller 5. The wireless transmissionfrom the communication section 36 to the controller communication module19 is sequentially performed at a predetermined time interval. Since thegame process is generally performed at a cycle of 1/60 sec.(corresponding to one frame time), data is transmitted at a cycle of ashorter time period. The communication section 36 of the controller 5outputs, to the controller communication module 19 of the game apparatus3, the operation data at intervals of 1/200 of a second, for example.

As described above, the controller 5 can transmit marker coordinatedata, acceleration data, angular rate data, and operation button data asoperation data representing operations performed thereon. In addition,the game apparatus 3 executes the game process using the operation dataas game inputs. Accordingly, by using the controller 5, the user canperform the game operation of moving the controller 5 itself, inaddition to conventionally general game operations of pressing operationbuttons. For example, it is possible to perform the operations oftilting the controller 5 to arbitrary attitudes, pointing the controller5 to arbitrary positions on the screen, and moving the controller 5itself.

Also, in the present example embodiment, the controller 5 is notprovided with any display means for displaying game images, but thecontroller 5 may be provided with a display means for displaying animage or suchlike to indicate, for example, a remaining battery level.

4. Configuration of the Terminal Device 7

Next, referring to FIGS. 8 to 11, the configuration of the terminaldevice 7 will be described. FIG. 8 is a plan view showing an externalconfiguration of the terminal device 7. FIG. 8( a) is a front view ofthe terminal device 7, FIG. 8( b) is a top view thereof, FIG. 8( c) is aright side view thereof, and FIG. 8( d) is a bottom view thereof. FIG. 9is a back view of the terminal device 7. FIG. 10 is a diagram showing auser holding the terminal device 7 in a landscape position.

As shown in FIG. 8, the terminal device 7 includes a housing 50generally in a horizontally-elongated rectangular plate shape. That is,it can also be said that the terminal device 7 is a tablet-typeinformation processing device. The housing 50 may have a curved surfaceor may have some protrusions, etc., as long as it is generally in aplate shape. The housing 50 is sized so that it can be held by the user.Thus, the user can hold and move the terminal device 7, and can changethe position in which the terminal device 7 is placed. The longitudinal(z-axis direction) length of the terminal device 7 is 100 to 150 [mm],for example, and is 133.5 [mm] in the present embodiment. The widthwise(x-axis direction) length of the terminal device 7 is 200 to 250 [mm],for example, and is 228.26 [mm] in the present embodiment. The thickness(the length in the y-axis direction) of the terminal device 7 is, forexample, about 15 to about 30 [mm] in a plate-shaped portion and about30 to about 50 [mm] including the thickest part, and is 23.6 (40.26 inthe thickest part) [mm] in the present embodiment. The weight of theterminal device 7 is about 400 to about 600 [g], and is 530 [g] in thepresent embodiment. Although the details will be described later, theterminal device 7 is configured so that it is easily held and operatedby the user even though it is such a relatively large terminal device(controller device) as described above.

The terminal device 7 includes an LCD 51 on the front surface (frontside) of the housing 50. The size of the screen of the LCD 51 is 5inches or larger, for example, and is herein 6.2 inches. The terminaldevice 7 of the present embodiment has such a configuration that it iseasily held and operated, and it is therefore easy to operate even if alarge LCD is provided. In other embodiments, the terminal device 7 maybe of a relatively small size with the provision of a smaller LCD 51.The LCD 51 is provided near the center of the surface of the housing 50.Therefore, the user can hold and move the terminal device 7 whilelooking at the screen of the LCD 51 by holding opposing end portions ofthe housing 50 with respect to the LCD 51, as shown in FIG. 10. WhileFIG. 10 shows an example in which the user holds the terminal device 7in a landscape position (in a horizontally-oriented direction) byholding left and right opposing end portions of the housing 50 withrespect to the LCD 51, the user can hold the terminal device 7 in aportrait position (in a vertically-oriented direction).

As shown in FIG. 8( a), the terminal device 7 includes a touch panel 52on the screen of the LCD 51 as an operation mechanism. In the presentembodiment, the touch panel 52 is a resistive-type touch panel. However,the touch panel is not limited to the resistive type, and may be a touchpanel of any type including, for example, a capacitive type, etc. Thetouch panel 52 may be of a single-touch type or a multi-touch type. Inthe present embodiment, a touch panel having the same resolution(detection precision) as the resolution of the LCD 51 is used as thetouch panel 52. However the resolution of the touch panel 52 does notalways need to coincide with the resolution of the LCD 51. While a touchpen 60 is usually used for making inputs on the touch panel 52, thepresent invention is not limited to using the touch pen 60, and an inputmay be made on the touch panel 52 with a finger of the user. The housing50 is provided with a hole 60 a for accommodating the touch pen 60 usedfor performing operations on the touch panel 52 (see FIG. 8( b)). Whilethe hole 60 a is provided on the upper surface of the housing 50 so thatthe touch pen 60 does not fall, it may be provided on the side surfaceor the bottom surface. Thus, since the terminal device 7 includes thetouch panel 52, the user can operate the touch panel 52 while moving theterminal device 7. That is, the user can move the screen of the LCD 51while directly (by means of the touch panel 52) making an input on thescreen.

As shown in FIG. 8, the terminal device 7 includes two analog sticks 53Aand 53B and a plurality of buttons (keys) 54A to 54M, as operationmechanisms (operation sections). The analog sticks 53A and 53B are eacha direction-specifying device. The analog sticks 53A and 53B are eachconfigured so that the movable member (stick portion) operated with afinger of the user can be slid in any direction (at any angle in the up,down, left, right and diagonal directions) with respect to the surfaceof the housing 50. That is, it is a direction input device which is alsocalled a slide pad. The movable member of each of the analog sticks 53Aand 53B may be of such a type that it is tilted in any direction withrespect to the surface of the housing 50. Since the present embodimentuse analog sticks of such a type that the movable members slide, theuser can operate the analog sticks 53A and 53B without significantlymoving the thumbs, and therefore operations can be made while thehousing 50 is held more firmly. When analog sticks of such a type thatthe movable members tilt are used as the analog sticks 53A and 53B, thedegree of input (the degree of tilt) is better perceived by the user,thus allowing the user to more easily perform precise operations.

The left analog stick 53A is provided on the left side of the screen ofthe LCD 51, and the right analog stick 53B is provided on the right sideof the screen of the LCD 51. Therefore, the user can make adirection-specifying input by using an analog stick with either the leftor the right hand. As shown in FIG. 10, the analog sticks 53A and 53Bare provided at such positions that the user can operate them whileholding the left and right portions of the terminal device 7 (the leftand right opposing end portions with respect to the LCD 51), andtherefore the user can easily operate the analog sticks 53A and 53B evenwhen holding and moving the terminal device 7.

The buttons 54A to 54L are operation mechanisms (operation sections) formaking predetermined inputs, and are keys that can be pressed. As willbe discussed below, the buttons 54A to 54L are provided at suchpositions that the user can operate them while holding the left andright portions of the terminal device 7 (see FIG. 10). Therefore, theuser can easily operate these operation mechanisms even when holding andmoving the terminal device 7.

As shown in FIG. 8( a), the cross button (direction-input button) 54Aand the buttons 54B to 54H and 54M, of the operation buttons 54A to 54L,are provided on the front surface of the housing 50. That is, thesebuttons 54A to 54H and 54M are provided at positions at which they canbe operated by the thumbs of the user (see FIG. 10).

The cross button 54A is provided on the left side of the LCD 51 andunder the left analog stick 53A. That is, the cross button 54A isprovided at such a position that it can be operated with the left handof the user. The cross button 54A has a cross shape, and is a buttonwith which it is possible to specify at least up, down, left and rightdirections.

The buttons 54B to 54D are provided on the lower side of the LCD 51.These three buttons 54B to 54D are provided at positions at which theycan be operated with either the left or the right hand. The terminaldevice 7 includes the power button 54M for turning ON/OFF the power ofthe terminal device 7. The power of the game apparatus 3 can be remotelyturned ON/OFF by operating the power button 54M. The power button 54M isprovided on the lower side of the LCD 51 as are the buttons 54B to 54D.The power button 54M is provided on the right side of the buttons 54B to54D. Thus, the power button 54M is provided at a position at which itcan be operated (easily operated) with the right hand. The four buttons54E to 54H are provided on the right side of the LCD 51 and under theright analog stick 53B. That is, the four buttons 54E to 54H areprovided at positions at which they can be operated with the right handof the user. Moreover, the four buttons 54E to 54H are provided on theupper, lower, left and right side (of the center position among the fourbuttons 54E to 54H). Therefore, with the terminal device 7, the fourbuttons 54E to 54H can also serve as buttons with which the userspecifies the up, down, left and right directions.

In the present embodiment, the analog sticks 53A and 53B are providedabove the cross button 54A and the buttons 54E to 54H. Here, the analogsticks 53A and 53B protrude beyond the cross button 54A and the buttons54E to 54H in the thickness direction (the y-axis direction). Therefore,if the positions of the analog stick 53A and the cross button 54A arereversed, the thumb of the user may inadvertently touch the analog stick53A when the user is operating the cross button 54A with the thumb. Asimilar problem occurs also when the positions of the analog stick 53Band the buttons 54E to 54H are reversed. In contrast, in the presentembodiment, since the analog sticks 53A and 53B are provided above thecross button 54A and the buttons 54E to 54H, the possibility that afinger may inadvertently touch the cross button 54A and the buttons 54Eto 54H when the user is operating the analog sticks 53A and 53B issmall. Thus, in the present embodiment, the possibility of erroneousoperations can be reduced, and it is possible to improve thecontrollability of the terminal device 7. Note however that in otherembodiments, the positions of the analog stick 53A and the cross button54A may be reversed and the positions of the analog stick 53B and thebuttons 54E to 54H may be reversed as necessary.

Here, in the present embodiment, some operation sections (the analogsticks 53A and 53B, the cross button 54A, and the three buttons 54E to54G) are provided on the left and right opposing sides of the displaysection (the LCD 51) and above the center of the housing 50 in theup/down direction (y-axis direction). When operating these operationsections, the user primarily holds a portion of the terminal device 7that is above the center thereof in the up/down direction. If the userholds the lower portion of the housing 50, the terminal device 7 to beheld becomes unstable, making it more difficult for the user to hold theterminal device 7. In contrast, in the present embodiment, whenoperating the operation section, the user primarily holds a portion ofthe terminal device 7 that is above the center thereof in the up/downdirection, and the housing 50 can be supported from the sides by thepalms. Therefore, the user can stably hold the housing 50 and it is madeeasier to hold the terminal device 7, thus making it easier to operatethe operation section. In other embodiments, operation sections areprovided, at least one on the left and one on the right of the displaysection, above the center of the housing 50. For example, only theanalog sticks 53A and 53B may be provided above the center of thehousing 50. For example, in a case in which the cross button 54A isprovided above the left analog stick 53A and the four buttons 54E to 54Hare provided above the right analog stick 53B, the cross button 54A andthe four buttons 54E to 54H may be provided above the center of thehousing 50.

In the present embodiment, a projecting portion (an eave portion 59) isprovided on the back side of the housing 50 (the side opposite to thefront surface where the LCD 51 is provided) (see FIGS. 8( c) and 9). Asshown in FIG. 8( c), the eave portion 59 is a ridge-shaped member whichprojects from the back surface of the generally plate-shaped housing 50.The projecting portion has such a height (thickness) that it can rest onfingers of the user holding the back surface of the housing 50. Theheight of the projecting portion is 10 to [mm], for example, and is16.66 [mm] in the present embodiment. The bottom surface of theprojecting portion may have an inclination of 45° or more (e.g., 60° ormore) with respect to the back surface of the housing 50 so that theprojecting portion easily rests on fingers of the user. As shown in FIG.8( c), the bottom surface of the projecting portion may have a largerinclination angle than the upper surface. As shown in FIG. 10, the usercan hold the terminal device 7 stably without getting tired even if theterminal device 7 has a relatively large size by holding the terminaldevice 7 with the eave portion 59 resting on the fingers (the eaveportion 59 being placed on the fingers). That is, the eave portion 59can be referred to as a supporting member by means of which the housing50 is supported by fingers, and can be referred to also as afinger-resting portion.

The eave portion 59 is provided above the center of the housing 50 withrespect to the up/down direction. The eave portion 59 is provided on thereverse side so as to generally correspond to the operation sections(the analog sticks 53A and 53B) which are provided on the front surfaceof the housing 50. That is, the projecting portion is provided so as toextend across an area on the reverse side including positionscorresponding to the operation sections which are provided respectivelyon the left side and on the right side of the display section.Therefore, when operating the operation section, the user can hold theterminal device 7 so as to support the eave portion 59 with the middlefingers or the ring fingers (see FIG. 10). Then, it is easier to holdthe terminal device 7, and it is easier to operate the operationsections. In the present embodiment, since the projecting portion has aneave-like shape extending in the left/right direction, the user can holdthe terminal device 7 with the middle fingers or the ring fingers placedalong the bottom surface of the projecting portion, making it easier tohold the terminal device 7. The eave portion 59 is not limited to theshape extending in the horizontal direction as shown in FIG. 9, as longas it is formed so that it (i.e., a projecting portion thereof) extendsin the left/right direction. In other embodiments, the eave portion 59may extend in a direction that is slightly inclined from the horizontaldirection. For example, the eave portion 59 may be provided so as to beinclined upwardly (or downwardly) from the left and right opposing endportions toward the center.

The present embodiment employs the eave portion 59 having a shape of aneave as the projecting portion formed on the back surface of the housingfor the purpose of providing engagement holes to be described below inthe eave portion 59, but the projecting portion may have any othersuitable shape. For example, in other embodiments, two projectingportions may be provided in the left and right opposing portions (withno projecting portion in the center of the left/right direction) on theback side of the housing 50 (see FIG. 29). In other embodiments, thecross-sectional shape (the shape along the cross section vertical to thex-axis direction) of the projecting portion may have an arched shape(which opens downward) so that the terminal device 7 can be more firmlysupported by the fingers of the user (so that the projecting portionmore firmly rests on the fingers).

The width of the projecting portion (the eave portion 59) in the up/downdirection may be of any value. For example, the projecting portion maybe formed so as to extend to the top side of the housing 50. That is,the upper surface of the projecting portion may be formed at the sameposition as the side surface on the upper side of the housing 50. Then,the housing 50 has a 2-tiered structure with the thin lower side and thethick upper side. As described above, the housing 50 may include adown-facing surface (the bottom surface of the projecting portion)formed in the left and right opposing portions of the back surface.Then, the user can easily hold the controller device with fingersabutting against this surface. While the “down-facing surface” may beprovided at any position on the back surface of the housing 50, it maybe located above the center of the housing 50.

As shown in FIGS. 8( a), 8(b) and 8(c), a first L button 54I and a firstR button 54J are provided respectively in the right and left opposingportions on the upper surface of the housing 50. In the presentembodiment, the first L button 54I and the first R button 54J areprovided on diagonally upper portions (the left upper portion and theright upper portion) of the housing 50. Specifically, the first L button54I is provided at the left end of the upper side surface of theplate-like housing 50 so that it is exposed on the upper left sidesurface (in other words, it is exposed on both the upper and left sidesurfaces). The first R button 54J is provided at the right end of theupper side surface of the housing 50, and is exposed on the upper rightside surface (in other words, it is exposed on both the upper and rightside surfaces). Thus, the first L button 54I is provided at such aposition that it can be operated with the left index finger of the user,and the first R button 54J is provided at such a position that it can beoperated with the right index finger of the user (see FIG. 10). In otherembodiments, the operation sections provided respectively in the leftand right portions of the upper surface of the housing 50 do not have tobe provided at the left end and the right end, and may be provided atpositions other than the end portions. The operation sections may beprovided respectively on the left and right side surfaces of the housing50.

As shown in FIGS. 8( c) and 9, a second L button 54K and a second Rbutton 54L are provided on the projecting portion (the eave portion 59).The second L button 54K is provided near the left end of the eaveportion 59. The second R button 54L is provided near the right end ofthe eave portion 59. Specifically, the second L button 54K is providedslightly toward the upper side in the left portion (the left portion asviewed from the front surface side) of the back surface of the housing50, and the second R button 54L is provided slightly toward the upperside in the right portion (the right portion as viewed from the frontsurface side) of the back surface of the housing 50. In other words, thesecond L button 54K is provided on the reverse side so as to (generally)correspond to the left analog stick 53A provided on the front surface,and the second R button 54L is provided on the reverse side so as to(generally) correspond to the right analog stick 53B provided on thefront surface. Thus, the second L button 54K is provided at a positionat which it can be operated with the left middle finger (or left indexfinger) of the user, and the second R button 54L is provided at aposition at which it can be operated with the right middle finger (orright index finger) of the user (see FIG. 10). The second L button 54Kand the second R button 54L are provided on the upper surface of theeave portion 59 as shown in FIG. 8( c). Therefore, the surfaces of thesecond L button 54K and the second R button 54L face upward (diagonallyupward). It is assumed that the middle fingers or the index fingers willgenerally move in the up/down direction when the user holds the terminaldevice 7, and it will be easier for the user to press the second Lbutton 54K and the second R button 54L if the button surfaces are facingupward.

As described above, in the present embodiment, operation sections (theanalog sticks 53A and 53B) are provided respectively on the left sideand the right side of the display section (the LCD 51) above the centerof the housing 50, and other operation sections (the second L button 54Kand the second R button 54L) are provided on the back side of thehousing 50 so as to generally correspond respectively to the operationsections. Thus, since the operation sections and the other operationsections are provided on the front side and on the back side of thehousing 50 so as to generally correspond to each other, the user canhold the housing 50 so as to sandwich the housing 50 from the front sideand from the back side when operating these operation sections. Whenoperating these operation sections, the user holds a portion of thehousing 50 that is above the center thereof in the up/down direction,and therefore the terminal device 7 can be held in the upper portionthereof and the terminal device 7 can be supported by the palms (seeFIG. 10). Thus, the user can stably hold the housing 50 in a state inwhich the user can operate at least four operation sections, and it istherefore possible to provide a controller device (the terminal device7) which can be easily held by the user and which has a goodcontrollability.

As described above, in the present embodiment, the user can easily holdthe terminal device 7 by holding the terminal device 7 with fingersabutting against the bottom surface of the projecting portion (the eaveportion 59). Since the second L button 54K and the second R button 54Lare provided on the upper surface of the projecting portion, the usercan easily operate these buttons in such a state as described above. Theuser can easily hold the terminal device 7 in the following manner, forexample.

That is, as shown in FIG. 10, the user can hold the terminal device 7with the ring fingers abutting against the bottom surface of the eaveportion 59 (the one-dot-chain line shown in FIG. 10) (so as to supportthe eave portion 59 with the ring fingers). Then, the user can operatethe four buttons (the first L button 54I, the first R button 54J, thesecond L button 54K and the second R button 54L) with the index fingersand the middle fingers. For example, in a case in which required gameoperations are relatively complicated and many buttons are to be used,it is possible to easily operate many buttons by holding the terminaldevice 7 as shown in FIG. 10. Since the analog sticks 53A and 53B areprovided above the cross button 54A and the buttons 54E to 54H, the usercan advantageously operate the analog sticks 53A and 53B with the thumbswhen relatively complicated operations are required. In FIG. 10, theuser holds the terminal device 7 with the thumbs abutting against thefront surface of the housing 50, the index fingers against the uppersurface of the housing 50, the middle fingers against the upper surfaceof the eave portion 59 on the back surface of the housing 50, the ringfingers against the bottom surface of the eave portion 59, and thelittle fingers against the back surface of the housing 50. Thus, theuser can firmly hold the terminal device 7 as if to wrap around thehousing 50 from four directions.

The user can also hold the terminal device 7 with the middle fingersabutting against the bottom surface of the eave portion 59. Then, theuser can easily operate two buttons (the second L button 54K and thesecond R button 54L) with the index fingers. For example, in a case inwhich required game operations are relatively simple and only a fewbuttons are to be used, the terminal device 7 may be held with themiddle fingers abutting against the bottom surface of the eave portion59. Then, since the user can hold the lower side of the housing 50 withtwo fingers (the ring finger and the little finger), it is possible tofirmly hold the terminal device 7.

In the present embodiment, the eave portion 59 is provided so that thebottom surface thereof is located between the analog sticks 53A and 53Band the cross button 54A and the four buttons 54E to 54H (so that it islocated on the lower side of the analog sticks 53A and 53B and above thecross button 54A and the four buttons 54E to 54H). Therefore, in a casein which the terminal device 7 is held with the ring fingers abuttingagainst the eave portion 59 (FIG. 10), the analog sticks 53A and 53B canbe easily operated with the thumbs, and in a case in which the terminaldevice 7 is held with the middle fingers abutting against the eaveportion 59, the cross button 54A and the four buttons 54E to 54H can beeasily operated with the thumbs. That is, in either of the two cases,the user can make a direction input operation while firmly holding theterminal device 7.

As described above, the user can also hold the terminal device 7 in aportrait position. That is, the user can hold the terminal device 7 in aportrait position by holding the top side or the lower side of theterminal device 7 with one hand. Thus, since the user can hold theterminal device 7 with one hand, it is possible to for example performan operation in which the terminal device 7 is held with one hand whilean input is made to the touch panel 52 with the other hand.

In a case in which the top side of the terminal device 7 is held, theuser can firmly hold the terminal device 7 by having fingers other thanthe thumbs abutting against the bottom surface of the eave portion 59.In the present embodiment, since the eave portion 59 extends in theleft/right direction, the user can abut fingers other than the thumbsagainst the eave portion 59 and firmly hold the terminal device 7,irrespective of the position along the top side of the terminal device 7at which the user holds the terminal device 7. That is, in a case inwhich the terminal device 7 is used in a portrait position, the eaveportion 59 can be used as a grip. On the other hand, in a case in whichthe bottom side of the terminal device 7 is held with one hand, the usercan operate the buttons 54B to 54D with that hand. Therefore, it ispossible for example to operate the buttons 54B to 54D with the handwith which the terminal device 7 is held while making inputs to thetouch panel 52 with the other hand, thereby allowing for moreoperations.

With the terminal device 7 of the present embodiment, since theprojecting portion (the eave portion 59) is provided on the backsurface, if the terminal device 7 is put down with the screen of the LCD51 (the front surface of the housing 50) facing up, the screen isslightly inclined. Therefore, the screen is more easily seen with theterminal device 7 put down. Input operations to the touch panel 52 aremore easily performed with the terminal device 7 put down. In otherembodiments, an additional projecting portion having generally the sameheight as the eave portion 59 may be formed on the back surface of thehousing 50. Then, with the screen of the LCD 51 facing up, the terminaldevice 7 can be put down so that the screen is horizontal with theprojecting portions in contact with the floor surface. The additionalprojecting portion may be a removable (or foldable) member. Then, theterminal device can be put down with the screen either slightly inclinedor with the screen horizontal. That is, in a case in which the terminaldevice 7 is put down and used, the eave portion 59 can be used as a legportion.

The buttons 54A to 54L are each assigned a function in accordance withthe game program. For example, the cross button 54A and the buttons 54Eto 54H may be used for direction-specifying operations, selectionoperations, etc., whereas the buttons 54B to 54E may be used for OKbutton operations, cancel button operations, etc. The terminal device 7may include a button for turning ON/OFF the power of the LCD 51, and abutton for performing a connection setting (pairing) with the gameapparatus 3.

As shown in FIG. 8( a), the terminal device 7 includes the markersection 55 including a marker 55A and a marker 55B on the front surfaceof the housing 50. The marker section 55 is provided on the upper sideof the LCD 51. The marker 55A and the marker 55B are each formed by oneor more infrared LEDs, as are the markers 6R and 6L of the marker device6. The infrared LEDs of the markers 55A and 55B are provided inside awindow portion that is transmissive to infrared light. The markersection 55 is used for the game apparatus 3 to calculate the movement,etc., of the controller 5, as is the marker device 6 described above.The game apparatus 3 can control the lighting of the infrared LEDs ofthe marker section 55.

The terminal device 7 includes a camera 56 as an image-capturingmechanism. The camera 56 includes an image-capturing element (e.g., aCCD image sensor, a CMOS image sensor, or the like) having apredetermined resolution, and a lens. As shown in FIG. 8, the camera 56is provided on the front surface of the housing 50 in the presentembodiment. Therefore, the camera 56 can capture an image of the face ofthe user holding the terminal device 7, and can capture an image of theuser playing a game while looking at the LCD 51, for example. In thepresent embodiment, the camera 56 is provided between the two markers55A and 55B.

The terminal device 7 includes a microphone 79 as a sound inputmechanism. A microphone hole 50 c is provided on the front surface ofthe housing 50. The microphone 79 is provided inside the housing 50behind the microphone hole 50 c. The microphone 79 detects sounds aroundthe terminal device 7 such as the voice of the user.

The terminal device 7 includes speakers 77 as sound output means. Asshown in FIG. 8( d), speaker holes 57 are provided in a lower portion ofthe front surface of the housing 50. The output sounds from the speakers77 are outputted from the speaker holes 57. In the present embodiment,the terminal device 7 includes two speakers, and the speaker holes 57are provided at the respective positions of each of the left speaker andthe right speaker. The terminal device 7 includes a knob 64 foradjusting the sound volume of the speakers 77. The terminal device 7includes a sound output terminal 62 for receiving a sound output sectionsuch as an earphone connected thereto. Although the sound outputterminal 62 and the knob 64 are provided on the upper side surface ofthe housing 50 considering the fact that the additional device isconnected to the lower side surface of the housing, they mayalternatively be provided on the left or right side surface or on thelower side surface.

The housing 50 includes a window 63 through which an infrared signalfrom an infrared communication module 82 is emitted to the outside ofthe terminal device 7. The window 63 is herein provided on the upperside surface of the housing so that the infrared signal is emitted in aforward direction of the user when the user holds the opposing sides ofthe LCD 51. In other embodiments, the window 63 may be provided at anyposition such as, for example, on the back surface of the housing 50.

The terminal device 7 includes an extension connector 58 via whichanother device can be connected to the terminal device 7. The extensionconnector 58 is a communication terminal for exchanging data(information) with another device connected to the terminal device 7. Inthe present embodiment, the extension connector 58 is provided on thelower side surface of the housing 50 as shown in FIG. 8( d). Theadditional device connected to the extension connector 58 may be anydevice, and may be, for example, a game-specific controller (gun-shapedcontroller, etc.) or an input device such as a keyboard. The extensionconnector 58 may be omitted if there is no need to connect an additionaldevice to terminal device 7. The extension connector 58 may include aterminal for supplying power to the additional device or a terminal forcharging.

In addition to the extension connector 58, the terminal device 7includes a charging terminal 66 for obtaining power from an additionaldevice. When the charging terminal 66 is connected to a charging stand(not shown), power is supplied from the charging stand to the terminaldevice 7. In the present embodiment, the charging terminal 66 isprovided on the lower side surface of the housing 50. Therefore, whenthe terminal device 7 and an additional device are connected to eachother, it is possible to supply power from one to the other, in additionto exchanging information therebetween, via the extension connector 58.Thus, with the provision of the charging terminal 66 around (on the leftand right opposing sides of) the extension connector 58, it is possibleto supply power, as well as exchange information, when the terminaldevice 7 and an additional device are connected to each other. Theterminal device 7 includes a charging connector, and the housing 50includes a cover portion 61 for protecting the charging connector. Thecharging connector can be connected to a charger 86 to be describedbelow, and power is supplied from the charger 86 to the terminal device7 when the charging connector is connected to the charger. Although thecharging connector (the cover portion 61) is provided on the upper sidesurface of the housing 50 in view of the fact that an additional deviceis connected to the lower side surface of the housing in the presentembodiment, it may be provided on the left and right side surfaces orthe lower side surface.

As shown in FIGS. 8( d) and 9, engagement holes 59 a and 59 b with whichtab portions of an additional device can engage are provided on thebottom surface of the projecting portion (the eave portion 59). Theengagement holes 59 a and 59 b are used when connecting an additionaldevice to the terminal device 7. That is, the additional device includestab portions which can engage with the engagement holes 59 a and 59 b,and when connecting the additional device to the terminal device 7, thetab portions are engaged with the engagement holes 59 a and 59 b,thereby securing the terminal device 7 and the additional device witheach other. Threaded holes may be further provided inside the engagementholes 59 a and 59 b so that the additional device can be securely fixedby screws. The projecting portion provided on the back surface of theterminal device 7 is herein the eave portion 59 having an eave-likeshape. That is, the eave portion 59 is provided so as to extend in theleft/right direction. As shown in FIG. 9, the engagement holes 59 a and59 b are provided near the center (with respect to the left/rightdirection) of the bottom surface of the eave portion 59. While thenumber of the engagement holes 59 a and 59 b provided on the bottomsurface of the eave portion 59 is not limited to any particular number,if there is one engagement hole, it may be provided at the center of theeave portion 59, and if there are a plurality of engagement holes, theymay be provided in left-right symmetry. Then, the additional device canbe stably connected while evenly maintaining the left-right balance. Ina case in which the engagement holes are provided near the center, thesize of the additional device can be reduced. Thus, the eave portion 59can be used as a member for engaging the additional device.

In the present embodiment, engagement holes 50 a and 50 b are providedon the bottom surface of the housing 50 as shown in FIG. 8( d).Therefore, in a case in which the additional device is connected to theterminal device 7, four tab portions are respectively engaged with fourengagement holes, thereby securing the terminal device 7 and theadditional device with each other. Thus, the additional device can bemore securely connected to the terminal device 7. Threaded holes may beprovided also inside the engagement holes 50 a and 50 b so that theadditional device can be screwed thereto. While screwing may be done atany position, the support portion of the additional device, which liesagainst the reverse surface of the housing 50, and the eave portion 59may be screwed together, for example. In other embodiments, theengagement holes provided in the housing may be in any arrangement.

The terminal device 7 includes a battery cover 67 which can be attachedto and removed from the housing 50. A battery (a battery 85 shown inFIG. 11) is placed inside the battery cover 67. In the presentembodiment, the battery cover 67 is provided on the back side of thehousing 50, below the projecting portion (the eave portion 59).

The housing 50 of the terminal device 7 includes holes 65 a and 65 bthrough which a strap cord can be tied to the terminal device 7. Asshown in FIG. 8( d), the holes 65 a and 65 b are provided on the bottomsurface of the housing 50 in the present embodiment. Two holes 65 a and65 b are provided in the present embodiment, one in the left portion andanother in the right portion of the housing 50. Specifically, the hole65 a is provided on the left side of the center of the bottom surface ofthe housing 50, and the hole 65 b is provided on the right side of thecenter of the bottom surface of the housing 50. The user can tie a strapto one of the holes 65 a and 65 b, and fasten the strap to the wrist ofthe user. Then, even if the user drops the terminal device 7 or if theterminal device 7 comes off the hand, the terminal device 7 is preventedfrom falling or hitting other objects. In the present embodiment, sincethe holes are provided both in the left and right portions, the user canconveniently fasten a strap to either hand.

With the terminal device 7 shown in FIGS. 8 to 11, the shape of eachoperation button, the shape of the housing 50, the number and thepositions of the components, etc., are merely illustrative, and thepresent invention can be realized with other shapes, numbers, andpositions.

Next, an internal configuration of the terminal device 7 will bedescribed with reference to FIG. 11. FIG. 11 is a block diagram showingan internal configuration of the terminal device 7. As shown in FIG. 11,in addition to the configuration shown in FIG. 8, the terminal device 7includes a touch panel controller 71, a magnetic sensor 72, theacceleration sensor 73, the gyroscope 74, a user interface controller(UI controller) 75, a codec LSI 76, the speakers 77, a sound IC 78, themicrophone 79, a wireless module 80, an antenna 81, an infraredcommunication module 82, a flash memory 83, a power supply IC 84, and abattery 85. These electronic components are mounted on an electroniccircuit board and accommodated in the housing 50.

The UI controller 75 is a circuit for controlling the input/output ofdata to/from various input/output sections. The UI controller 75 isconnected to the touch panel controller 71, an analog stick 53 (theanalog sticks 53A and 53B), an operation button 54 (the operationbuttons 54A to 54L), the marker section 55, the magnetic sensor 72, theacceleration sensor 73, and the gyroscope 74. The UI controller 75 isconnected to the codec LSI 76 and the extension connector 58. The powersupply IC 84 is connected to the UI controller 75, and power is suppliedto various sections via the UI controller 75. The built-in battery 85 isconnected to a power supply IC 84 to supply power. The charger 86 or acable with which power can be obtained from an external power source canbe connected to the power supply IC 84 via a charging connector, and theterminal device 7 can receive power supply from or be charged by anexternal power source using the charger 86 or the cable. The terminaldevice 7 may be charged by attaching the terminal device 7 to a cradle(not shown) having a charging function. That is, although not shown inthe drawings, a cradle with which power can be obtained from an externalpower supply can be connected to the power supply IC 84 via the chargingterminal 66, and the terminal device 7 can receive power supply from orbe charged by an external power supply using the cradle.

The touch panel controller 71 is a circuit connected to the touch panel52 for controlling the touch panel 52. The touch panel controller 71generates touch position data of a predetermined format based on signalsfrom the touch panel 52, and outputs it to the UI controller 75. Thetouch position data represents, for example, the coordinates of aposition on the input surface of the touch panel 52 at which an input ismade. The touch panel controller 71 reads a signal from the touch panel52 and generates touch position data at a rate of once per apredetermined amount of time. Various control instructions for the touchpanel 52 are outputted from the UI controller 75 to the touch panelcontroller 71.

The analog stick 53 outputs, to the UI controller 75, stick datarepresenting the direction and the amount of slide (or tilt) of thestick portion operated with a finger of the user. The operation button54 outputs, to the UI controller 75, operation button data representingthe input status of each of the operation buttons 54A to 54L (e.g.,whether it is pressed).

The magnetic sensor 72 detects an azimuth by sensing the size anddirection of the magnetic field. Azimuth data representing the detectedazimuth is outputted to the UI controller 75. Control instructions forthe magnetic sensor 72 are outputted from the UI controller 75 to themagnetic sensor 72. While there are sensors using an MI (magneticimpedance) element, a fluxgate sensor, a Hall element, a GMR (giantmagneto-resistive) element, a TMR (tunnel magneto-resistance) element,an AMR (anisotropic magneto-resistive) element, etc., the magneticsensor 72 may be any sensor as long as it is possible to detect anazimuth. Strictly speaking, in a place where there is a magnetic fieldother than the geomagnetic field, the obtained azimuth data does notrepresent the azimuth. Nevertheless, if the terminal device 7 moves, theazimuth data changes, and it is therefore possible to calculate thechange in the attitude of the terminal device 7.

The acceleration sensor 73 is provided inside the housing 50 fordetecting the magnitude of the linear acceleration along each of thedirections of the three axes (the x, y and z axes shown in FIG. 8( a)).Specifically, the acceleration sensor 73 detects the magnitude of thelinear acceleration along each of the axes, where the x axis lies in thelongitudinal direction of the housing 50, the y axis lies in thedirection vertical to the surface of the housing 50, and the z axis liesin the width direction of the housing 50. Acceleration data representingthe detected acceleration is outputted to the UI controller 75. Controlinstructions for the acceleration sensor 73 are outputted from the UIcontroller 75 to the acceleration sensor 73. While the accelerationsensor 73 is assumed to be a capacitive-type MEMS-type accelerationsensor, for example, in the present embodiment, other types ofacceleration sensors may be employed in other embodiments. Theacceleration sensor 73 may be an acceleration sensor for 1-axis or2-axis detection.

The gyroscope 74 is provided inside the housing 50 for detecting angularrates about the three axes, i.e., the x-axis, the y-axis and the z-axis.Angular rate data representing the detected angular rates is outputtedto the UI controller 75. Control instructions for the gyroscope 74 areoutputted from the UI controller 75 to the gyroscope 74. The number andcombination of gyroscopes used for detecting angular rates about threeaxes may be arbitrary, and the gyroscope 74 may be formed by a 2-axisgyroscope and a 1-axis gyroscope, as is the gyroscope 48. The gyroscope74 may be a gyroscope for 1-axis or 2-axis detection.

The UI controller 75 outputs, to the codec LSI 76, operation dataincluding touch position data, stick data, operation button data,azimuth data, acceleration data, and angular rate data received fromvarious components described above. If another device is connected tothe terminal device via the extension connector 58, data representing anoperation performed on the other device may be further included in theoperation data.

The codec LSI 76 is a circuit for performing a compression process ondata to be transmitted to the game apparatus 3, and an expansion processon data transmitted from the game apparatus 3. The LCD 51, the camera56, the sound IC 78, the wireless module 80, the flash memory 83, andthe infrared communication module 82 are connected to the codec LSI 76.The codec LSI 76 includes a CPU 87 and an internal memory 88. While theterminal device 7 does not itself perform game processes, the terminaldevice 7 may execute a minimal program for the management thereof andfor the communication. When the terminal device 7 is started up, aprogram stored in the flash memory 83 is read out to the internal memory88 and executed by the CPU 87 upon power-up. Some area of the internalmemory 88 is used as the VRAM for the LCD 51.

The camera 56 captures an image in response to an instruction from thegame apparatus 3, and outputs the captured image data to the codec LSI76. Control instructions for the camera 56, such as an image-capturinginstruction, are outputted from the codec LSI 76 to the camera 56.Camera 56 can also record video. That is, the camera 56 can repeatedlycapture images and repeatedly output the image data to the codec LSI 76.

The sound IC 78 is a circuit connected to the speakers 77 and themicrophone 79 for controlling input/output of sound data to/from thespeakers 77 and the microphone 79. That is, when sound data is receivedfrom the codec LSI 76, the sound IC 78 outputs sound signals obtained byperforming D/A conversion on the sound data to the speakers 77 so thatsound is outputted from the speakers 77. The microphone 79 detectssounds propagated to the terminal device 7 (the sound of the user,etc.), and outputs sound signals representing such sounds to the soundIC 78. The sound IC 78 performs A/D conversion on the sound signals fromthe microphone 79 to output sound data of a predetermined format to thecodec LSI 76.

The infrared communication module 82 emits an infrared signal to performinfrared communication with another device. Here, for example, theinfrared communication module 82 has the function of performing infraredcommunication in accordance with the IrDA standard and the function ofoutputting an infrared signal (control signal) to control the television2.

The codec LSI 76 transmits, as terminal operation data, image data fromthe camera 56, sound data from the microphone 79 and operation data fromthe UI controller 75 to the game apparatus 3 via the wireless module 80.In the present embodiment, the codec LSI 76 performs a compressionprocess similar to that of the codec LSI 27 on the image data and thesound data. The terminal operation data and the compressed image dataand sound data are outputted, as transmit data, to the wireless module80. The antenna 81 is connected to the wireless module 80, and thewireless module 80 transmits the transmit data to the game apparatus 3via the antenna 81. The wireless module 80 has a similar function tothat of the terminal communication module 28 of the game apparatus 3.That is, the wireless module 80 has a function of connecting to awireless LAN by a scheme in conformity with the IEEE 802.11n standard,for example. The transmitted data may be encrypted as necessary or maynot be encrypted.

As described above, the transmit data transmitted from the terminaldevice 7 to the game apparatus 3 includes operation data (the terminaloperation data), image data, and sound data. In a case in which anotherdevice is connected to the terminal device 7 via the extension connector58, data received from the other device may be further included in thetransmit data. The codec LSI 76 may transmit data received throughinfrared communication by the infrared communication module 82, to thegame apparatus 3, with the data being included in the transmit data,where appropriate.

As described above, compressed image data and sound data are transmittedfrom the game apparatus 3 to the terminal device 7. These data arereceived by the antenna 81 (reception section) and transferred to thecodec LSI 76 via the wireless module 80. The codec LSI 76 expands thereceived image data and sound data. The expanded image data is outputtedto the LCD 51, and images are displayed on the LCD 51. That is, thecodec LSI 76 (the CPU 87) displays the received image data on thedisplay section. The expanded sound data is outputted to the sound IC78, and the sound IC 78 outputs sounds from the speakers 77.

In a case in which control data is included in data received from thegame apparatus 3, the codec LSI 76 and the UI controller 75 give controlinstructions to various sections in accordance with the control data. Asdescribed above, the control data is data representing controlinstructions for the components of the terminal device 7 (the camera 56,the touch panel controller 71, the marker section 55, sensors 62 to 64,and the infrared communication module 82 in the present embodiment). Inthe present embodiment, control instructions represented by control datamay be instructions to activate the operation of the components ordeactivate (stop) the operation thereof. That is, components that arenot used in a game may be deactivated in order to reduce the powerconsumption, in which case it is ensured that data from the deactivatedcomponents are not included in the transmit data transmitted from theterminal device 7 to the game apparatus 3. For the marker section 55,which is an infrared LED, the control can be done simply by turningON/OFF the power supply thereto.

Furthermore, the game apparatus 3 is capable of controlling output ofthe infrared communication module 82, thereby controlling the operationof the television 2. Specifically, the game apparatus 3 outputs aninstruction (control data as mentioned above) to the terminal device 7,thereby causing the infrared communication module 82 to output aninfrared signal corresponding to a control command for controlling thetelevision 2. In response to this instruction, the codec LSI 76 causesthe infrared communication module 82 to output an infrared signalcorresponding to the control command. Here, the television 2 includes aninfrared light reception section capable of receiving the infraredsignal. By the infrared light reception section receiving the infraredsignal outputted by the infrared communication module 82, the television2 operates in accordance with the infrared signal. Note that theinstruction from the game apparatus 3 may indicate the pattern of theinfrared signal, or when the terminal device 7 has the infrared signalpattern stored therein, the game apparatus 3 may provide an instructionto indicate the pattern.

While the terminal device 7 includes operation mechanisms such as thetouch panel 52, an analog stick 53 and the operation button 54, asdescribed above, in other embodiments, other operation mechanisms may beincluded instead of, or in addition to, these operation mechanisms.

While the terminal device 7 includes the magnetic sensor 72, theacceleration sensor 73 and the gyroscope 74 as sensors for calculatingmovement of the terminal device 7 (including the position and theattitude thereof, or changes in the position and the attitude thereof),it may only include one or two of these sensors in other embodiments. Inother embodiments, other sensors may be included instead of, or inaddition to, these sensors.

While the terminal device 7 includes the camera 56 and the microphone79, it may not include the camera 56 and the microphone 79 or it mayinclude only one of them in other embodiments.

While the terminal device 7 includes the marker section 55 as aconfiguration for calculating the positional relationship between theterminal device 7 and the controller 5 (the position and/or attitude,etc., of the terminal device 7 as seen from the controller 5), it maynot include the marker section 55 in other embodiments. In otherembodiments, the terminal device 7 may include other mechanisms as aconfiguration for calculating the positional relationship. For example,in other embodiments, the controller 5 may include a marker section, andthe terminal device 7 may include an image-capturing element. Moreover,in such a case, the marker device 6 may include an image-capturingelement, instead of an infrared LED.

(Configuration for Connecting the Game System to an External Device)

FIG. 12 is a block diagram illustrating the connection relationshipbetween the game system 1 and an external device. As shown in FIG. 12,in the game system 1, the game apparatus 3 is capable of connecting toan external device 91 via a network 90. The network 90 is an arbitrarycommunication network such as the Internet. The external device 91 is aWeb server or another terminal device (e.g., in the case of acommunication game, a destination game apparatus) capable ofcommunicating with the game apparatus 3. Note that there may be morethan one external device 91, and the game apparatus 3 may communicatewith a plurality of external devices. The game apparatus 3 communicateswith the network 90, and acquires Web pages, various applications, gamedata, and so on, from the external device 91 via the network 90. Thegame apparatus 3 outputs the acquired image information, or any images,etc., generated on the basis of the acquired information, to theterminal device 7 and/or the television 2. The game system 1 is capableof displaying various images on two display devices.

5. General Operations in the Game System

Next, general operations performed in the game system 1 will bedescribed. In the present example embodiment, the game system 1 iscapable of executing various applications, including a game program(game application), performing operations on the television 2 (anddisplaying a television program guide), and displaying Web pages.Moreover, by operations shown below, the game system 1 allows auser-friendly operation for switching between a state where thetelevision is used in the game system 1 (the television 2 displaysimages generated in the game system 1) and another state where thetelevision is used for a purpose other than in the game system 1 (thetelevision 2 displays images different from those generated in the gamesystem 1).

FIG. 13 is a state transition diagram representing transitions betweenoperation states of the game system 1. As shown in FIG. 13, when thegame system 1 (game apparatus 3) is booted, the game system 1 is firstbrought into an initial state. The “initial state” is a state after bootof the game system 1, in which the terminal device 7 displays apredetermined operation image. Here, the predetermined operation imagemay be any image allowing the user of the terminal device 7 to performan operation. In the present example embodiment, the terminal device 7displays any one of the following images to be described later:application selection image (hereinafter, indicated as “app selectionimage”); device control image; and browser image, as a predeterminedoperation image. Note that in another example embodiment, the terminaldevice 7 may display more than one of the three images at the same time(e.g., by dividing its display area). Moreover, in the following, anyoperation image displayed in the initial state may also be referred toas a “main operation image” for the purpose of being distinguished fromany operation image displayed in an interrupted state to be describedlater.

Note that the “initial state” refers to a state after boot of the gameapparatus 3, where an application can be selected, and is not limited toa state immediately after the boot. For example, a predetermined imagemay be displayed for a predetermined period of time immediately afterthe boot, and thereafter, the initial state may be brought about.Alternatively, the initial state may occur again after the execution andshutdown of a predetermined application.

In the initial state, the terminal device 7 displays a predeterminedmain operation image, and the terminal device is also used for anoperation in the game system. Specifically, the terminal device 7, whichis an operating device, includes an operation section for outputtingoperation data (terminal operation data) representing an operation onthe operating device. Moreover, the terminal device 7 includes areception section (antenna 81 and wireless module 80) for receiving animage outputted from the game apparatus 3, and display section (LCD 51)for displaying the received image. On the other hand, the game apparatus3 includes an acquisition section for acquiring the operation data, andgenerates the main operation image in the initial state. In addition,the game apparatus 3 outputs the main operation image to the terminaldevice 7.

Note that, in the present example embodiment, the initial state includes“menu state”, “first television operation state”, and “browser state” asshown in FIG. 13. The menu state is a state (mode) in which variousapplications can be selected and executed. The first televisionoperation state is a state (mode) in which the television 2 can beoperated. The browser state is a state (mode) in which Web pages can bedisplayed. As will be described in detail later, the terminal device 7displays the app selection image (FIG. 17) in the menu state, the devicecontrol image (FIG. 19) in the first television operation state, and thebrowser image (FIG. 21) in the browser state. Note that the appselection image is an operation image for selecting an applicationexecutable by the game apparatus 3. The device control image is anoperation image for manipulating (controlling) the television 2. Thebrowser image is an operation image including a Web page acquired viathe network 90. Note that in the initial state, the terminal device 7always displays a predetermined operation image (main operation image),and therefore, in another example embodiment, only one of theaforementioned images may always be displayed in the initial state.

Furthermore, when the game system 1 is booted into an initial state, thegame system 1 takes one of the aforementioned three states (i.e., menustate, first television operation state, and browser state). Note thatthe state of the game system 1 immediately after boot may be determinedarbitrarily. In the present example embodiment, the state of the gamesystem 1 upon the last shutdown is set as the state immediately afterboot. In addition, as will be described in detail later, the game system1 can transition from one of the three states to any of the other twostates in accordance with a predetermined change operation (see FIG.13). Specifically, by the change operation, the user can readily switchoperation images to be displayed on the terminal device 7 among the appselection image, the device control image, and the browser image.

Here, in the menu state of the initial state, the user can select andexecute one of various applications, including game programs, using theterminal device 7. In addition, in the first television operation stateof the initial state, the user can operate the television 2 using theterminal device 7. Specifically, in the initial state, on the basis ofterminal operation data, the game apparatus 3 determines whether or nota selection instruction has been provided (step S14 to be describedlater), and also determines whether or not a control instruction hasbeen provided (step S23 to be described later). The selectioninstruction is an instruction to select an application from one or moreapplications executable by the game apparatus 3, one of the applicationsbeing a game application. In addition, the control instruction is aninstruction to control the television 2. When the selection instructionhas been provided, the game apparatus 3 executes the selectedapplication, and outputs an image generated by executing the applicationto either the terminal device 7 or the television 2, or both (step S15to be described later). Moreover, when the control instruction has beenprovided, the game apparatus 3 controls the television 2 in accordancewith the control instruction (step S24 to be described later).

Thus, in the present example embodiment, the terminal device 7 displaysan operation image (main operation image) in the initial state, so thatthe user can operate the game system 1 while viewing the screen of theterminal device 7, and the television 2 can display other images such asa television program. Accordingly, the user can use the game system 1while watching a television program. For example, while watching atelevision program on the screen of the television 2, the user can viewan operation image on the terminal device 7 and select an application tobe executed. Moreover, in the initial state, the user can operate thetelevision 2 using the terminal device 7. The user can operate both thegame system 1 and the television 2 using the terminal device 7, whichmakes it easy to perform an operation to switch between the state wherethe television is used in the game system 1 and the state where thetelevision is used for a purpose other than in the game system 1. Forexample, in the present example embodiment, a television program guide(electronic program guide) is displayed as a device control image, sothat, by simply using the terminal device 7, the user can perform aseries of operations for booting the game system 1 to display theprogram guide on the terminal device 7, and selecting a desiredtelevision program from the displayed program guide to display theprogram on the television 2. In this manner, in the present exampleembodiment, the terminal device 7 makes it possible to enhanceuser-friendliness in the aforementioned switching operation.

Note that in the present example embodiment, the game apparatus 3 causesthe terminal device 7 to display either the app selection image or thedevice control image at one time, but in another example embodiment, theterminal device 7 may display the app selection image and the devicecontrol image at the same time. As a result, the user can freely provideboth the selection instruction and the control instruction withoutperforming an operation (a change operation as mentioned above) totransition between the state (menu state) where the selectioninstruction can be provided and the state (first television operationstate) where the control instruction can be provided. Alternatively, inanother example embodiment, the game apparatus 3 may allow the user toprovide the control instruction in the menu state where the appselection image is displayed, or may allow the user to provide theselection instruction in the first television operation state where thedevice control image is displayed. This also makes it possible for theuser to freely perform the two instructions without performing thechange operation.

Furthermore, in the initial state, the game apparatus 3 determineswhether or not a display instruction to display a Web page has beenprovided (steps S17 and S26 to be described later). When the displayinstruction has been provided, the game system 1 transitions to thebrowser state (see FIG. 13). In this case, the game apparatus 3 outputsa browser image, which is generated on the basis of data acquired fromthe network 90 via an external communication section (networkcommunication module 18) communicating with the network 90, to eitherthe terminal device 7 or the television 2, or both (steps S31 and S35 tobe described later). Specifically, in response to the displayinstruction, the game apparatus 3 causes the terminal device 7 and/orthe television 2 to display a browser image including the Web page. Inthis manner, in the present example embodiment, the user can browse theWeb page in the initial state.

Furthermore, when an operation to select an application is performedusing the terminal device 7 in the menu state, the game apparatus 3executes the selected application. Specifically, the game apparatus 3generates a main operation image (an app selection image as mentionedabove) including images (e.g., icons) representing applicationsexecutable by the game apparatus 3, and outputs the generated image tothe terminal device 7. Thereafter, when the main operation image isdisplayed, and any application to be executed is selected, the selectedapplication is executed. As a result, the game system 1 transitions toan app execution state, as shown in FIG. 13. Moreover, the gameapparatus 3 outputs an image generated by executing the application toeither the terminal device 7 or the television 2, or both. Note that one(or both) of the two display devices to which the image is outputted maybe determined arbitrarily in accordance with the application to beexecuted.

In the app execution state, when a predetermined interruption operationis performed on the terminal device 7, the game system 1 transitions tothe interrupted state (see FIG. 13). The interrupted state is a statewhere the use of the application being executed is interrupted. When thepredetermined interruption operation is performed on the terminal device7 while the application is being executed, the game apparatus 3generates and outputs a predetermined operation image to the terminaldevice 7 (step S44 to be described later). The predetermined operationimage may be any image allowing the user of the terminal device 7 toperform an operation. In the present example embodiment, the terminaldevice 7 displays a setting image (FIG. 23) or remote control image(FIG. 24) to be described later as the predetermined operation image.Note that in another example embodiment, the terminal device 7 maydisplay only one operation image or may display the aforementioned twoimages at the same time. The operation image displayed in theinterrupted state will be referred to below as the “sub-operationimage”.

The interrupted state includes a setting state and a second televisionoperation state. In the present example embodiment, upon transition fromthe app execution state to the interrupted state, the game system 1 isbrought into the setting state. The setting state is a state where thesetting image is displayed. As will be described in detail later, thesetting image is an operation image allowing various setting operationsto be performed on the game system 1. While the terminal device 7displays the predetermined sub-operation image in the interrupted state,in another example embodiment, the terminal device 7 may display onlyone of the two images without switching to the other during theinterrupted state.

In the present example embodiment, when the predetermined operation isperformed in the setting state, the game system 1 transitions to thesecond television operation state. The second television operation stateis a state where the remote control image is displayed. As will bedescribed in detail later, the remote control image is an operationimage allowing an operation to be performed on the television 2, andincluding button images or suchlike which represent operations on thetelevision 2. The second television operation state is a state where thetelevision 2 can be operated using the terminal device 7. Note that inthe second television operation state, when a predetermined operation isperformed, the game system 1 transitions from the second televisionoperation state to the setting state. That is, the user can freely makea transition between the setting state and the second televisionoperation state by a predetermined operation (see FIG. 13).

In this manner, in the present example embodiment, the user can operatethe television 2 in the interrupted state. Specifically, with theterminal device 7 displaying the sub-operation image, the game apparatus3 determines whether or not a control instruction to control thetelevision 2 has been provided, on the basis of terminal operation data.When the control instruction has been provided, the television 2 iscontrolled in accordance with the control instruction. Accordingly, inthe present example embodiment, when an application is being used in thegame system 1, the user can operate the television 2 without stoppingthe execution of the application. As a result, it is rendered possibleto more readily perform an operation of switching between the statewhere the television is used in the game system 1 and the state wherethe television is used for a purpose other than in the game system 1.For example, by using the terminal device 7, the user can readilyperform an operation to watch a television program while executing anapplication and afterward perform an operation to resume theapplication.

Furthermore, in the present example embodiment, the interrupted stateallows a termination instruction to terminate an application beingexecuted and a resumption instruction to resume the execution of anapplication. Specifically, when the resumption instruction is providedin the interrupted state, the game apparatus 3 resumes the execution ofan application. As a result, the game system 1 transitions to the appexecution state (see FIG. 13). In addition, when the terminationinstruction is provided in the interrupted state, the game apparatus 3ends the execution of an application. As a result, the game system 1transitions to the initial state (see FIG. 13). Note that in the abovecase, while in the present example embodiment, the game systemtransitions to the menu state, as shown in FIG. 13, in another exampleembodiment, the game system 1 may transition to the first televisionoperation state or the browser state. Moreover, in another exampleembodiment, when terminating an application in the interrupted state,the user may be able to select the state to which the system istransitioned from among the three initial states. In this manner, in thepresent example embodiment, the user can both resume and terminate theexecution of an application in the interrupted state.

6. Details of the Processing in the Game System

Next, the information processing executed in the present game systemwill be described in detail. First, various data for use in theinformation processing will be described. FIG. 14 is a diagramillustrating a storage area of the main memory (the external main memory12 or the internal main memory 11 e) in the game apparatus 3. As shownin FIG. 14, the main memory of the game apparatus 3 has provided thereina main program region 100, an application program region 101, anacquired data region 103, and a process data region 107. Note that inaddition to the data shown in FIG. 14, the main memory has storedtherein data, such as image data and sound data, to be used in theinformation processing. Moreover, the storage area shown in FIG. 14 maybe provided in part in another storage device (e.g., the flash memory17) accessible by the game apparatus 3.

The main program region 100 has stored therein a main program formanaging transitions between the aforementioned states in the gamesystem 1. In the present example embodiment, the CPU 10 executes themain program, thereby performing each step in the flowchart shown inFIG. 15 and other flowcharts. The main program is a program forperforming processes to be described later, including a menu process, atelevision process, a browser process, and an interruption-time process,but in another example embodiment, individual programs for performingthese processes respectively may be separately stored in the mainmemory. The main program is read in whole or in part from apredetermined storage device at an appropriate time after the power-onof the game apparatus 3, and then stored to the main memory. Note thatthe predetermined storage device may be any storage accessible by thegame apparatus 3, and it is the flash memory 17 in the present exampleembodiment. Moreover, the main program may be acquired from the opticaldisc 4 or a device external to the game apparatus 3 (e.g., via theInternet), rather than from the flash memory 17.

The application program region 101 is intended for storage of variousapplications (programs) executable by the game apparatus 3. Theapplication program region 101 has one or more applications storedtherein, and in the present example embodiment, the one or moreapplications include a game program 102. The game program 102 is read inwhole or in part from the optical disc 4 at an appropriate time afterthe power-on of the game apparatus 3, and then stored to the mainmemory. Note that the game program 102 may be acquired from the flashmemory 17 or a device external to the game apparatus 3 (e.g., via theInternet), rather than from the optical disc 4. Note that any otherapplication to be included in the application program region 101 may beacquired from an arbitrary storage accessible by the game apparatus 3,as in the case of the game program 102.

The acquired data region 103 is intended for storage of various dataacquired from the terminal device 7, etc., by the game apparatus 3. Inthe present example embodiment, the acquired data region 103 is used forstoring terminal operation data 104, camera image data 105, andmicrophone sound data 106. Note that in addition to the data shown inFIG. 14, the acquired data region 103 may have stored therein, forexample, controller operation data acquired from the controller 5.

The terminal operation data 104 is data representing the player'soperation on the terminal device 7. The terminal operation data 104 istransmitted by the terminal device 7, acquired by the game apparatus 3,and then stored to the main memory. Note that the game apparatus 3 iscapable of communicating with a plurality of terminal devices, andacquiring operation data from each of the terminal devices. In the casewhere there are a plurality of terminal devices, terminal operation datatransmitted by each of the terminal devices is individually stored tothe main memory. The main memory may have the terminal operation datastored for each terminal device up to a predetermined number of piecescounted from the latest piece (the last acquired piece).

The terminal operation data 104 includes angular rate data, accelerationdata, operation button data, stick data, touch position data, andazimuth data. The angular rate data is data representing angular ratesdetected by the gyroscope 74. In the present example embodiment, theangular rate data represents angular rates about three axes, x-, y-, andz-axes, shown in FIG. 8, but in another example embodiment, the data mayrepresent an angular rate about each of any one or more axes. Theacceleration data is data representing acceleration (accelerationvector) detected by the acceleration sensor 73. In the present exampleembodiment, the acceleration data represents three-dimensionalacceleration whose components are acceleration values associated withthe directions of three axes, x-, y-, and z-axes, shown in FIG. 9, butin another example embodiment, the data may represent accelerationassociated with any one or more directions. The gyroscope 74 and theacceleration sensor 73 are example inertial sensors. The game apparatus3 can calculate the attitude of the terminal device 7 on the basis ofthe results of detection by the inertial sensors (angular rate data andacceleration data).

Furthermore, the operation button data is data representing an inputstate of each of the operation buttons 54A to 54L provided on theterminal device 7. Specifically, the operation button data indicateswhether any of the operation buttons 54A to 54L has been pressed. Thestick data is data representing the direction and the amount of sliding(or tilting) of the stick portion of each of the analog sticks 53A and53B. The amount and the direction may be represented by, for example,two-dimensional coordinates or a two-dimensional vector. The touchposition data is data representing a position (touch position) on theinput screen of the touch panel 52 at which an input has been made. Inthe present example embodiment, the touch position data represents acoordinate value in a two-dimensional coordinate system which indicatesa position on the input screen. Note that in the case where the touchpanel 52 is multi-touch, the touch position data may represent aplurality of touch positions. The azimuth data is data representing anazimuth detected by the magnetic sensor 72. In the present exampleembodiment, the azimuth data represents a predetermined azimuth (e.g.,north) with respect to the terminal device 7. Note that in a place wherethere is a magnetic field in addition to the geomagnetic field, theazimuth data does not strictly indicate an absolute azimuth (such asnorth). However, the azimuth data indicates the direction of theterminal device 7 relative to the direction of the magnetic field in theplace, and therefore, even in the case as above, it is possible tocalculate the attitude or the change in the attitude of the terminaldevice 7 based on the azimuth data.

Note that the terminal operation data 104 may only include any one ofthe aforementioned data so long as an operation on the terminal device 7is represented. In addition, in the case where the terminal device 7 hasanother input means (e.g., a touch pad, or an imaging means as providedin the controller 5), the terminal operation data 104 may include datarepresenting an operation on such an input means. Note that in the casewhere the motion of the terminal device 7 itself is used as a gameoperation as in the present example embodiment, the terminal operationdata 104 may include data representing information related to the motionof the terminal device 7 (data whose value changes in accordance withthe motion), as in the case of the acceleration data, the angular ratedata, and the azimuth data. For example, the information related to themotion of the terminal device 7 is information concerning the position,attitude, change in position (speed), change in attitude (angular rate),acceleration, and angular acceleration of the terminal device 7.

The camera image data 105 is data representing an image (pickup image)obtained by the camera 56 of the terminal device 7. The camera imagedata 105 is image data obtained by the codec LSI 27 decompressingcompressed image data transmitted by the terminal device 7, and theinput/output processor 11 a stores the camera image data 105 to the mainmemory. The microphone sound data 106 is data representing sound(microphone sound) detected by the microphone 79 of the terminal device7. The microphone sound data 106 is obtained by the codec LSI 27decompressing compressed sound data transmitted by the terminal device7, and the input/output processor 11 a stores the microphone sound data106 to the main memory.

The process data region 107 is intended for storage of various data tobe used in the information processing (such as that shown in FIG. 15) bythe game apparatus 3. In the present example embodiment, the processdata region 107 stores initial state data 108, program guide data 109,home page data 110, save data 111, and so on. Note that in addition tothe data shown in FIG. 14, the process data region 107 has storedtherein various data for use in executing applications.

The initial state data 108 is data representing an initial state of thegame system 1 immediately after boot, the initial state being one of thethree initial states. The initial state data 108 may be set arbitrarily,but in the present example embodiment, the initial state is determinedin accordance with the final state of the game system 1 upon the lastshutdown (power-off or sleep).

The program guide data 109 is data for use in generating a televisionprogram guide to be displayed on the terminal device 7 upon transitionto the first television operation state. In the present exampleembodiment, data for a program guide is acquired in a timely manner froma predetermined server via the network 90, and stored to the main memoryas program guide data 109. In the present example embodiment, when thegame system 1 transitions to the first television operation state, theterminal device 7 initially displays a program guide generated on thebasis of the program guide data 109.

The home page data 110 is data for use in generating a home page (a Webpage displayed at the start of the browser function) to be displayed onthe terminal device 7 upon transition to the browser state. In thepresent example embodiment, data for a preset home page is acquired in atimely manner from a predetermined server via the network 90, and storedto the main memory as home page data 110. In the present exampleembodiment, when the game system 1 transitions to the browser state, theterminal device 7 initially displays a Web page image generated on thebasis of the home page data 110.

The save data 111 is application save data for the game program 102 andso on. As will be described in detail later, when the execution of anapplication is terminated, data representing the content of the game andgame parameters for resuming the game is stored to the main memory assave data 111.

Next, a flow of the information processing to be executed by the gamesystem 1 (game apparatus 3) will be described in detail with referenceto FIGS. 15 to 24. FIG. 15 is a flowchart illustrating a flow of aninitial process performed by the game apparatus 3. The initial processis a process to be performed in response to boot of the game system(game apparatus 3). Specifically, when the game apparatus 3 is poweredon, the CPU 10 of the game apparatus 3 executes a boot program stored inan unillustrated boot ROM, thereby initializing each unit, including themain memory. A main program stored in the flash memory 17 is loaded tothe main memory, and the CPU 10 starts executing the main program. Theprocessing shown in the flowchart of FIG. 15 is performed uponcompletion of the above processing.

Note that the processing in each step of the flowcharts shown in thefigures is merely illustrative, and if similar results can be achieved,the processing order of the steps may be changed. In addition, values ofvariables and thresholds to be used in determination steps are alsomerely illustrative, and other values may be used appropriately.Furthermore, while the processing in each step of the flowcharts isdescribed herein as being executed by the CPU 10, the processing in thesteps of the flowcharts may be performed in part by a processor otherthan the CPU 10 or by specialized circuits.

First, in step S1, the CPU 10 sets one of the three states (menu state,first television operation state, and browser state) as the firstinitial state to be taken. Here, the initial state may be setarbitrarily, and in the present example embodiment, the initial stateupon the next boot is set on the basis of the final state of the gamesystem 1 upon the last shutdown. Specifically, when the game system 1 isshut down (powered off or brought into sleep mode), data representingthe state at that time is stored to the flash memory 17. Then, in stepS1, the CPU 10 reads that data from the flash memory 17, and sets theinitial state on the basis of the state represented by the data.Concretely, when the data represents any one of the three initialstates, the state represented by the data is set as the initial state.Alternatively, when the data represents the app execution state or theinterrupted state, the menu state is set as the initial state. Datarepresenting the initial state thus set is stored to the main memory asinitial state data. In this manner, in the present example embodiment,the state of the game system 1 upon the last shutdown is set as theinitial state, and therefore, the last user of the game system 1 canrestart using the game system 1 in an appropriate state. Following stepS1, the process of step S2 is performed.

Note that in another example embodiment, the initial state may be set byanother method. For example, the initial state may be set to a statepreviously designed by the user. This also allows the user tostart/restart using the game system 1 in an appropriate state, as in thepresent example embodiment. In addition, for example, when the gameapparatus 3 is capable of identifying users, the initial state may beset to states corresponding to users of the game system 1. As a result,even when the game system 1 is used by a plurality of users, each usercan start/restart using the game system 1 in an appropriate state forthat user. Note that the “states corresponding to users” may be statesset on the basis of the states of the game system 1 upon the lastshutdown by the users or may be states designated by the users.Moreover, the method for identifying users may be arbitrary. Forexample, the CPU 10 may identify users by prompting the users to entertheir IDs and/or passwords upon boot. Alternatively, for example, theCPU 10 may identify users by using images of their faces taken by thecamera 56 of the terminal device 7. Concretely, it is possible toidentify the users by registering the images of the users' faces inadvance and comparing the images taken by the camera 56 to theregistered images.

In steps S2 and S3, program guide data and home page data are acquiredin advance. Here, these data may be acquired at any arbitrary time, andin the present example embodiment, they are acquired upon boot of thegame system 1. Note that in another example embodiment, these data maybe acquired at the time of transition to a state where a program guideor a home page is to be displayed (i.e., the first television operationstate or the browser state). In addition, as the home page data, dataacquired in the previous use of the game system 1 may be saved using thecache function of a general browser.

In step S2, the CPU 10 acquires a program guide from a predeterminedserver (external device 91) via the network 90. Specifically, the CPU 10first generates an acquisition request for the program guide, andtransmits the acquisition request to the server via the network 90. Theserver may be any server capable of providing program guide data, e.g.,a Web server having program guide data stored therein. In response tothe acquisition request, the server transmits program guide data to thegame apparatus 3 via the network 90. The program guide data transmittedfrom the server is stored to the flash memory 17 (or the main memory).Note that in another example embodiment, program guide image datagenerated from the data acquired from the server may be stored insteadas program guide data 109. Following step S2, the process of step S3 isperformed.

In step S3, the CPU 10 acquires data for a Web page (home page) to beaccessed at the start of the browser function. Specifically, the CPU 10first generates an acquisition request for the Web page, and transmitsthe acquisition request to the predetermined server via the network 90.The server may be any information processing apparatus capable ofproviding a preset Web page as mentioned above. In response to theacquisition request, the server transmits Web page data to the gameapparatus 3 via the network 90. The Web page data transmitted from theserver is stored to the flash memory 17 (or the main memory). Note thatin another example embodiment, Web page image data generated from thedata acquired from the server may be stored instead as home page data110. The CPU 10 ends the initial process after step S3.

After the initial process, a process in accordance with the initialstate set in step S1 is executed. Specifically, a menu process to bedescribed later is executed when the initial state is the menu state, atelevision process to be described later is executed when the initialstate is the first television operation state, and a browser process tobe described later is executed when the initial state is the browserstate. Hereinafter, the menu process, the television process, and thebrowser process will be described in detail.

(Menu Process)

FIG. 16 is a flowchart illustrating a flow of the menu process performedby the game apparatus 3. First, in step S11, the CPU 10 acquires variousdata from the terminal device 7. Here, at least terminal operation datais acquired from the terminal device 7, and in the present exampleembodiment, camera image data and microphone sound data are furtheracquired. The terminal device 7 repeatedly transmits these data(terminal operation data, camera image data, and microphone sound data)to the game apparatus 3, and the game apparatus 3 sequentially receivesthe data. In the game apparatus 3, the terminal communication module 28sequentially receives the data, and the codec LSI 27 sequentiallysubjects the camera image data and the microphone sound data todecompressing processing. Then, the input/output processor 11 asequentially stores the terminal operation data, the camera image data,and the microphone sound data to the main memory. The CPU 10 reads thesedata at appropriate times. Following step S11, the process of step S12is performed.

In step S12, the CPU 10 displays an app selection image on the terminaldevice 7. Specifically, the CPU 10 generates an app selection image, andoutputs (transmits) the generated app selection image to the terminaldevice 7. The app selection image may be any image which allows the userto select an application executable by the game apparatus 3.Hereinafter, with reference to FIG. 17, the app selection image in thepresent example embodiment will be described.

FIG. 17 is a diagram illustrating an example app selection imagedisplayed on the terminal device 7. As shown in FIG. 17, the appselection image includes an app area 121, a notification area 123, atelevision icon 125, and a browser icon 126. The app selection imageshown in FIG. 17 is a menu image for selecting an application to beexecuted by the game apparatus 3.

In FIG. 17, the app area 121 is an area in which one or more app icons122 are displayed. The app icon 122 is an image (icon) representing anapplication executable by the game apparatus 3. In the present exampleembodiment, by an operation of selecting the app icon 122, theapplication represented by the app icon 122 is executed. Note that inFIG. 17, the app area 121 can be horizontally scrolled in accordancewith a predetermined scrolling operation, and horizontal scrollingchanges the app icon 122 to be displayed. The notification area 123 isan area in which a notification to the user is displayed. Thenotification area 123 may display any information, including, forexample, presentation of a new application, and notifications aboutvarious services related to the game system 1. Note that the gameapparatus 3 may access a predetermined server via the network 90 at anappropriate time and acquire notification information, thereby updatingthe contents of the notification.

The television icon 125 is an image representing an instruction totransition to the first television operation state. As will be describedin detail later, in the present example embodiment, by an operation ofselecting the television icon 125 (a change operation as describedabove), the game system 1 is caused to transition to the firsttelevision operation state. The browser icon 126 is an imagerepresenting an instruction to transition to the browser state. As willbe described in detail later, in the present example embodiment, by anoperation of selecting the browser icon 126 (a change operation asdescribed above), the game system 1 is caused to transition to thebrowser state. Specifically, in the menu state, by performing anoperation to select the television icon 125 or the browser icon 126, theuser can readily cause the game system 1 to transition to the firsttelevision operation state or the browser state.

In step S12, the CPU 10 and the GPU 11 b collaborate to read data,including image data for various icons, which is used for generating anapp selection image, from the VRAM 11 d, the data being read is used togenerate the app selection image. The generated app selection image isstored to the VRAM 11 d. Moreover, the CPU 10 outputs the app selectionimage to the terminal device 7. Concretely, the CPU 10 transfers datafor the app selection image stored in the VRAM 11 d to the codec LSI 27,and the codec LSI 27 subjects the data to predetermined compressionprocessing. The terminal communication module 28 transmits thecompressed image data to the terminal device 7 via the antenna 29. Inthe terminal device 7, the wireless module 80 receives the image datatransmitted by the game apparatus 3, and the codec LSI 76 performspredetermined decompressing processing on the received image data. Thedecompressed image data is outputted to the LCD 51. As a result, the appselection image is displayed on the LCD 51. Moreover, in step S12, sounddata may be transmitted to the terminal device 7 along with the imagedata, so that the speakers 77 of the terminal device 7 output sound.Following step S12, the process of step S13 is performed.

In step S13, a television image is displayed on the television 2.Specifically, the CPU 10 generates a predetermined television image, andoutputs the generated television image to the television 2. The contentof the television image to be displayed in the menu state may bearbitrary. For example, the television image may be an image related toan application executable by the game apparatus 3 (e.g., a title imageor demonstration video of a game application) or may be an image to bedisplayed in the notification area 123. Note that it is envisioned that,in the menu state, the television 2 is used for a purpose other than inthe game system 1 (e.g., to display a television program), andtherefore, the content of the television image may be intended forauxiliary use.

In step S13, the CPU 10 and the GPU 11 b collaborate to read data foruse in television image generation from the VRAM 11 d, and generate atelevision image. The generated television image is stored to the VRAM11 d. In addition, the CPU 10 outputs the television image to thetelevision 2. Concretely, the CPU 10 transfers television image datastored in the VRAM 11 d to the AV-IC 15. In response to this, the AV-IC15 outputs the television image data to the television 2 via the AVconnector 16. As a result, the television image is displayed on thetelevision 2. Moreover, in step S13, sound data may be outputted to thetelevision 2 along with the image data, so that sound is outputted fromthe speakers 2 a of the television 2. Following step S13, the process ofstep S14 is performed.

In step S14, the CPU 10 determines whether or not a selectioninstruction has been provided. The selection instruction is aninstruction to select an application to be executed. The selectioninstruction may be provided by any operation, and in the present exampleembodiment, the selection instruction is provided by an operation oftouching an app icon 122 displayed on the terminal device 7.Specifically, the CPU 10 reads touch data in the terminal operation data104 from the main memory, and determines whether or not any app icon 122displayed on the terminal device 7 has been touched, on the basis of thetouch data. Note that in another example embodiment, for example, theselection instruction may be provided using a cursor displayed on theterminal device 7. Specifically, the CPU 10 may determine that an appicon 122 has been selected when a predetermined operation was performedwith the cursor being placed on the app icon 122 after having been movedin accordance with a direction input operation on the terminal device 7(e.g., a direction input operation with the analog stick 53). When theresult of the determination of step S14 is affirmative, the process ofstep S15 is performed. On the other hand, when the result of thedetermination of step S14 is negative, the process of step S16 isperformed.

In step S15, the CPU 10 executes the application selected by theselection instruction. Specifically, the CPU 10 reads the selectedapplication from a storage (e.g., the optical disc 4 or the flash memory17) having the application stored therein, and then stores theapplication to the main memory. Thereafter, the execution of theapplication stored in the main memory is started. After step S15, theCPU 10 ends the menu process, and executes an app execution process. Asa result, the game system 1 transitions to the app execution state.

Although not shown, information processing in accordance with theapplication is executed in the app execution process. The content of theapplication to be executed may be arbitrary, and information processingto be performed by executing the application may also be arbitrary. TheCPU 10 outputs an image generated by executing the application to eitherthe terminal device 7 or the television 2, or both. Note that adetermination as to whether to display the generated image on either theterminal device 7 or the television 2, or both, may be made arbitrarily.For example, the determination may be made by the application or theuser. When the generated image is displayed only on the terminal device7, the television 2 can display a television program, so that the usercan use the application on the game system 1 while viewing thetelevision program.

On the other hand, in step S16, the CPU 10 determines whether or not totransition to the first television operation state. Specifically, theCPU 10 determines whether or not a change operation has been performedto transition from the menu state to the first television operationstate. The change operation may be any operation, and in the presentexample embodiment, it is an operation of touching the television icon125 displayed on the terminal device 7. Specifically, the CPU 10 readsthe touch data in the terminal operation data 104 from the main memory,and determines whether or not the television icon 125 displayed on theterminal device 7 has been touched, on the basis of the touch data. Whenthe result of the determination of step S16 is affirmative, the CPU 10ends the menu process, and performs a television process to be describedlater. On the other hand, when the result of the determination of stepS16 is negative, the process of step S17 is performed.

Note that in the present example embodiment, various instructions(operations) in, for example, steps S14, S16, S17, S23, S25, S26, S33,S34, S43, S45, S47, S50, and S52 are provided using the touch panel 52.Specifically, the CPU 10 (in concert with the GPU 11 b) can generateimages representing various instructions (e.g., the selectioninstruction, the control instruction, and instructions for the changeoperations), and can output and display the images on the terminaldevice 7. Furthermore, when such an image is selected (touched) on thetouch panel 52, an instruction as mentioned above is determined to havebeen provided. Thus, the user can provide various instructions by easyand intuitive operations using the touch panel.

Note that in another example embodiment, various instructions may beprovided by other operations. For example, the instructions may beprovided in whole or in part by operations using the cursor displayed onthe terminal device 7. Specifically, predetermined instructions may beprovided by performing predetermined operations with the cursor beingplaced on images (e.g., icons) representing the instructions afterhaving been moved in accordance with a direction input operation on theterminal device 7. Alternatively, for example, the instructions may beprovided in whole or in part by operations of pressing any of thebuttons 54A to 54M of the terminal device 7. More specifically, aninstruction (the change operation as mentioned above) to transition fromone state to another of the three initial states may be provided by anoperation of pressing predetermined buttons (e.g., the second L button54K and the second R button 54L) of the terminal device 7. Specifically,each pressing of a predetermined button of the terminal device 7 changesthe app selection image, the device control image, and the browser imageone by one in sequence. Alternatively, for example, the instructions maybe provided in whole or in part by operations of moving the terminaldevice 7. Note that the movement of the terminal device 7 can becalculated (estimated) using the results of detection by the inertialsensors (the acceleration sensor 73 and the gyroscope 74) and themagnetic sensor 72.

In step S17, the CPU 10 determines whether to transition to the browserstate. Concretely, the CPU 10 determines whether or not a displayinstruction to display a Web page has been provided, i.e., a changeoperation to transition from the menu state to the browser state hasbeen performed. The change operation may be any operation, and in thepresent example embodiment, it is an operation of touching the browsericon 126 displayed on the terminal device 7. Specifically, the CPU 10reads the touch data in the terminal operation data 104 from the mainmemory, and determines whether or not the browser icon 126 displayed onthe terminal device 7 has been touched, on the basis of the touch data.When the result of the determination of step S17 is affirmative, the CPU10 ends the menu process, and performs the browser process to bedescribed later. On the other hand, when the result of the determinationof step S17 is negative, the process of step S11 is performed again.Thereafter, until the result of the determination of step S14, S16, orS17 turns out to be affirmative, the menu process is repeatedlyperformed.

(Television Process)

FIG. 18 is a flowchart illustrating a flow of the television processperformed by the game apparatus 3. The television process is performedwhen the first television operation state is determined as the initialstate or when a change operation to transition from the menu state orthe browser state to the first television operation state is performed.First, in step S21, the CPU 10 acquires various data from the terminaldevice 7. The process of step S21 is the same as that of step S11.Following step S21, the process of step S22 is performed.

In step S22, the CPU 10 displays a device control image on the terminaldevice 7. Specifically, the CPU 10 generates a device control image, andoutputs (transmits) the generated device control image to the terminaldevice 7. The device control image may be any operation imagerepresenting an operation on the television 2. In the present exampleembodiment, a television program guide image is acquired from theexternal device 91 via the network 90, and displayed as the devicecontrol image. Hereinafter, referring to FIG. 19, the device controlimage in the present example embodiment will be described.

FIG. 19 is a diagram illustrating an example device control imagedisplayed on the terminal device 7. As shown in FIG. 19, the devicecontrol image includes a program guide area 130, operation button images136 to 138, a menu icon 124, and the browser icon 126. The devicecontrol image shown in FIG. 19 is a program guide image, includingimages representing control instructions to the television 2.

In FIG. 19, the program guide area 130 is an area in which to display aprogram guide generated on the basis of data acquired from apredetermined server via the network 90. Here, the program guide area130 includes channel images 131, a program information area 132, atuning instruction button 133, a scrollbar 134, a date operation button136, a detail display button 137, and a setting button 138. The channelimages 131 are images representing television channels and programnames. In FIG. 19, the channel images 131 are vertically arranged, andthe user can scroll up and down to display desired channel images 131 bymoving a knob 135 of the scrollbar 134. The program information area 132is an area in which to display information about a television programrepresented by any given one of the displayed channel images 131 (theinformation being about the content, cast, etc., of the program). Here,the given channel image 131 is a channel image 131 touched by the user.

The tuning instruction button 133 is an image representing a controlinstruction to the television 2, concretely, a control instruction totune in to the channel represented by the given channel image. Note thatthe image representing the control instruction is not limited to thetuning instruction button 133, and for example, the device control imagemay include a button representing a control instruction to turn on oroff the television 2, adjust the volume of the television 2, or switchbetween inputs to the television 2.

The date operation button 136 is a button for providing an instructionto change the date (and time) of a program guide to be displayed. Thedetail display button 137 is a button for providing an instruction todisplay more detailed information than the displayed content in theprogram information area 132. The setting button 138 is a button forproviding an instruction to display a setting screen for program guidesand the terminal device 7. Note that in addition to the above, theterminal device 7 may display, for example, an image for program guidesearch, and an image for displaying a list of favorite programs. Byusing the program guide image, the user can browse the content oftelevision programs, and operate the television 2 to tune in to adesired channel. In the present example embodiment, the televisionprogram guide is displayed in the first television operation state, andtherefore, the user can conveniently change the channel of thetelevision 2 while viewing the program guide.

The menu icon 124 is an image representing an instruction to transitionto the menu state. As will be described in detail later, in the presentexample embodiment, by an operation of selecting the menu icon 124 (thechange operation as mentioned above), the game system 1 is caused totransition to the menu state. The browser icon 126 is the same as thatshown in FIG. 16. Accordingly, in the first television operation state,by an operation of selecting the menu icon 124 or the browser icon 126,the user can readily cause the game system 1 to transition to the menustate or the browser state.

In step S22, the CPU 10 and the GPU 11 b collaborate to read data foruse in device control image generation from both the main memory and theVRAM 11 d, and a device control image is generated using the data beingread. In the present example embodiment, any image to be displayed inthe program guide area 130 is generated using the program guide data 109stored in the main memory. Accordingly, in the present exampleembodiment, even immediately after transition to the first televisionoperation state, the device control image can be quickly displayed onthe terminal device 7. Moreover, the generated device control image isstored to the VRAM 11 d. In addition, the CPU 10 outputs the devicecontrol image to the terminal device 7. The process of outputting anddisplaying an image on the terminal device 7 is the same as in step S12.As a result, the device control image is displayed on the LCD 51.Moreover, in step S22, sound data may be transmitted to the terminaldevice 7 along with the image data, so that the speakers 77 of theterminal device 7 output sound. Following step S22, the process of stepS23 is performed.

In step S23, it is determined whether or not a control instruction hasbeen provided. The control instruction is an instruction to control thetelevision 2. The operation for providing the control instruction may bearbitrary, and in the present example embodiment, the controlinstruction is provided by an operation of touching a button (tuninginstruction button 133) displayed on the terminal device 7, representingthe control instruction. Specifically, the CPU 10 reads the touch datain the terminal operation data 104 from the main memory, and determineswhether or not the button displayed on the terminal device 7 has beentouched, on the basis of the touch data. When the result of thedetermination of step S23 is affirmative, the process of step S24 isperformed. On the other hand, when the result of the determination ofstep S23 is negative, the process of step S24 is skipped, and theprocess of step S25 is performed.

In step S24, the CPU 10 controls the television 2 in accordance with thecontrol instruction. Here, the method for the game system 1 to controlthe television 2 is arbitrary. In the present example embodiment, thetelevision 2 can be controlled by a first method in which the gameapparatus 3 outputs a control signal to the television 2 via theterminal device 7, and a second method in which the game apparatus 3outputs a control signal directly to the television 2.

In the first method, the CPU 10 instructs the terminal device 7 totransmit a control signal indicating the content of control according tothe control instruction. In accordance with the instruction, theterminal device 7 outputs the control signal to control the television2. Concretely, the CPU 10 outputs control data representing theinstruction to the terminal device 7. Note that in the present exampleembodiment, the flash memory 17 or the main memory has stored thereindata representing various control signals for causing the television 2to perform various operations. From the various control signals, the CPU10 chooses data representing a control signal according to the controlinstruction, and outputs the data to the terminal device 7 as controldata. Upon reception of the control data, the terminal device 7 outputsthe control signal represented by the control data. In the presentexample embodiment, an infrared signal is outputted as the controlsignal. Specifically, the codec LSI 76 of the terminal device 7 causesthe infrared communication module 82 to output an infrared signalcorresponding to the control signal represented by the control data. Bythe infrared light reception section of the television 2 receiving theinfrared signal, the television 2 operates in accordance with thecontrol instruction.

In the second method, the CPU 10 outputs a control signal indicating thecontent of control according to the control instruction to thetelevision 2. Specifically, from various control signals for causing thetelevision 2 to perform various operations, the CPU 10 chooses a controlsignal (e.g., an HDMI signal) according to the control instruction. Thechosen control signal is outputted to the television 2 via the AVconnector 16. This also causes the television 2 to operate in accordancewith the control instruction.

Note that the format of the control signal (infrared signal) to controlthe television 2 may vary between models of the television 2.Accordingly, the game apparatus 3 may prestore control signals informats corresponding to a plurality of models. In this case, the gameapparatus 3 may select a format corresponding to the television 2 at apredetermined time (e.g., at the time of initial setting), andthereafter, use control signals in the selected format. Thus, the gameapparatus 3 is compatible with a plurality of models of television.

Note that in the present example embodiment, the game apparatus 3 maycontrol the television 2 using either or both of the two methods.Following step S24, the process of step S25 is performed.

In step S25, the CPU 10 determines whether or not to transition to themenu state. Specifically, the CPU 10 determines whether or not a changeoperation has been performed to transition from the first televisionoperation state to the menu state. The change operation may bearbitrary, and in the present example embodiment, it is an operation oftouching the menu icon 124 displayed on the terminal device 7.Specifically, the CPU 10 reads the touch data in the terminal operationdata 104 from the main memory, and determines whether or not the menuicon 124 displayed on the terminal device 7 has been touched, on thebasis of the touch data. When the result of the determination of stepS25 is affirmative, the CPU 10 ends the television process, and performsthe menu process. On the other hand, when the result of thedetermination of step S25 is negative, the process of step S26 isperformed.

In step S26, the CPU 10 determines whether or not to transition to thebrowser state. The process of step S26 is the same as the process ofstep S17. When the result of the determination of step S26 isaffirmative, the CPU 10 ends the television process, and performs thebrowser process to be described later. On the other hand, when theresult of the determination of step S26 is negative, the process of stepS27 is performed.

In step S27, the CPU 10 performs other television processing. The othertelevision processing refers to processing in the television process,excluding steps S21 to S26. For example, the CPU 10 performs theprocessing of changing the content of display in the program guide area130 in accordance with selection operations on the operation buttonimage 136 to 138, or the processing of scrolling in accordance with ascrolling operation on the scrollbar 134 to display desired channelimages 131. Moreover, in the present example embodiment, assuming thatthe television 2 is used to display a television program in the firsttelevision operation state, the CPU 10 does not generate nor output anytelevision image in step S27. However, in another example embodiment, atelevision image may be generated and outputted. For example, theaforementioned device control image representing a program guide may beoutputted and displayed on the television 2 as the television image.Thus, the user can browse the program guide on a larger screen.

Following step S27, the process of step S21 is performed again.Thereafter, until the result of the determination of step S25 or S26turns out to be affirmative, the television process is repeatedlyperformed.

The television process allows the user to browse a program guide by theterminal device 7 (step S22), and perform operations on the television 2(steps S23 and S24). In addition, by a change operation, the user canreadily cause the first television operation state to transition to themenu state or the browser state (step S25 or S26).

Note that in another example embodiment, upon transition from the menustate or the browser state to the first television operation state,inputs to the television 2 may be switched (so that a televisionbroadcast is displayed). Specifically, when the television 2 can beswitched between a first mode in which images inputted from the gameapparatus 3 are displayed and a second mode in which televisionbroadcast images are displayed, the CPU 10 may control the television 2to change its mode to the second mode in accordance with the display ofthe terminal device 7 switching from the app selection image (or thebrowser image) to the device control image. Concretely, at the start ofthe television process, the CPU 10 may perform input switching controlof the television 2 such that the television 2 is set to the second modeto display a television broadcast. As a result, upon transition to thefirst television operation state, the user does not perform anyoperation to switch between inputs to the television 2, resulting inmore simplified user operations. Note that the input switching controlmay be performed by either the first or second method described inconjunction with step S24.

Furthermore, in another example embodiment, upon transition from thefirst television operation state to the menu state or the browser state,inputs to the television 2 may be switched (so that images from the gameapparatus 3 are displayed). Specifically, when the television 2 can beswitched between the first mode in which images inputted from the gameapparatus 3 are displayed and the second mode in which televisionbroadcast images are displayed, the CPU 10 may control the television 2to change its mode to the first mode in accordance with the display ofthe terminal device 7 switching from the device control image to the appselection image (or the browser image). Concretely, when the result ofthe determination of step S16 is affirmative, the CPU 10 may performinput switching control of the television 2 such that the television 2is set to the first mode to display images from the game apparatus 3. Inaddition, when the result of the determination of step S17 isaffirmative, the CPU 10 may similarly perform input switching control ofthe television 2 such that the television 2 is set to the first mode. Asa result, upon transition from the first television operation state toanother state, the user does not perform any operation to switch betweeninputs to the television 2, resulting in more simplified useroperations. Note that the input switching control may be performed byeither the first or second method described in conjunction with stepS24.

Note that depending on the model of the television 2, it might be onlypossible to provide instructions to sequentially switch between aplurality of inputs (e.g., instructions to sequentially switch betweenfive inputs numbered from “1” to “5” one at a time), and might not bepossible to provide an instruction to select the second mode to displaya television broadcast. In such a case, the number of input switchinginstructions to be provided to switch between the first mode and thesecond mode may be registered in advance, and the CPU 10 may provide thesame number of input switching instructions as the registered number.For example, in the case where a television broadcast is displayed on“input 1”, images from the game apparatus 3 are displayed on “input 3”and the number of input switching instructions to be provided to switchfrom “input 1” to “input 3” is two, the game apparatus 3 may store “2”as the number of input switching instructions. Note that at the initialsetting of the game system 1, the user may be prompted to register aninput number corresponding to the game apparatus 3, so that the gameapparatus 3 can calculate the number of input switching instructions.

(Browser Process)

FIG. 20 is a flowchart illustrating a flow of the browser processperformed by the game apparatus 3. First, in step S21, the terminaldevice 7 displays a Web page image as a browser image. Specifically, theCPU 10 generates a browser image, and outputs (transmits) the generatedbrowser image to the terminal device 7. The browser image may be anyimage including a Web page acquired via the network 90. Hereinafter,referring to FIG. 21, the browser image in the present exampleembodiment will be described.

FIG. 21 is a diagram illustrating an example browser image displayed onthe terminal device 7. As shown in FIG. 21, the browser image includes aWeb page display area 141, a scrollbar 142, a back button 143, a forwardbutton 144, a keyword area 145, a search button 146, and a remotecontrol display button 147. The browser image further includes the menuicon 124 and the television icon 125. The browser image shown in FIG. 21is an operation image for displaying a Web page and allowing operationson the Web page.

In FIG. 21, the Web page display area 141 is an area in which to displaya Web page image generated on the basis of data acquired from thenetwork 90. The Web page displayed in the Web page display area 141 canbe scrolled up and down by moving a knob of the scrollbar 142 up anddown. The back button 143 is an image representing an instruction todisplay one Web page previous to the current page. The forward button144 is an image representing an instruction to display one Web page nextto the current page. The keyword area 145 is an area in which an enteredsearch keyword is displayed. The search button 146 is an imagerepresenting an instruction to perform a Web page search using a searchkeyword. The buttons 143 to 146 may function in the same manner as inconventional and general browsers.

The remote control display button 147 is an image representing aninstruction to display the remote control image (FIG. 24). The remotecontrol image will be described in detail later in conjunction with aninterruption-time process. The menu icon 124 and the television icon 125are the same as those shown in FIGS. 16 and 18. Accordingly, in thebrowser state, by performing an operation to select the menu icon 124 orthe television icon 125, the user can readily cause the game system 1 totransition to the menu state or the first television operation state.

In step S31, the CPU 10 and the GPU 11 b collaborate to read data foruse in browser image generation from both the main memory and the VRAM11 d, and generate a browser image using the data being read. Note thatin step S1 of the present example embodiment, any image to be displayedin the Web page display area 141 is generated using the home page data110 stored in the main memory. Accordingly, in the present exampleembodiment, even immediately after transition to the browser state, thebrowser image can be quickly displayed on the terminal device 7.Moreover, the generated browser image is stored to the VRAM 11 d. Inaddition, the CPU 10 outputs the browser image to the terminal device 7.The process of outputting and displaying an image on the terminal device7 is the same as in step S12. As a result, the browser image isdisplayed on the LCD 51. Note that the CPU 10 may output at least aportion of the browser image to the terminal device 7. For example, theCPU 10 may cause the terminal device 7 to display an enlarged image of aportion of the browser image. As a result, the browser image can bedisplayed in a more easy-to-view format even on the terminal device 7,which generally has a smaller screen than the television 2. Moreover, instep S31, sound data may be transmitted to the terminal device 7 alongwith the image data, so that the speakers 77 of the terminal device 7output sound. Following step S31, the process of step S32 is performed.

In step S32, the CPU 10 acquires various data from the terminal device7. The process of step S32 is the same as the process of step S11.Following step S32, the process of step S33 is performed.

In step S33, the CPU 10 determines whether or not to transition to themenu state. The process of step S33 is the same as the process of stepS25. When the result of the determination of step S33 is affirmative,the CPU 10 ends the browser process, and performs the menu process. Onthe other hand, when the result of the determination of step S33 isnegative, the process of step S34 is performed.

In step S34, the CPU 10 determines whether or not to transition to thefirst television operation state. The process of step S34 is the same asthe process of step S16. When the result of the determination of stepS34 is affirmative, the CPU 10 ends the browser process, and performsthe television process. On the other hand, when the result of thedetermination of step S34 is negative, the process of step S35 isperformed.

In step S35, the CPU 10 performs other browser processing. The otherbrowser processing refers to processing in the browser process,excluding steps S31 to S34. For example, the CPU 10 performs processingcorresponding to the buttons 143 to 146 chosen by selection operationsor the processing of scrolling a displayed Web page in accordance with ascrolling operation on the scrollbar 142. In addition, when a selectionoperation is performed on link information on a Web page in step S35,the CPU 10 performs the processing of acquiring a linked Web page or theprocessing of playing a video, if any, on the Web page. In step S35,various processing may be executed as in general browsers.

Furthermore, the CPU 10 may perform the processing of generating andoutputting a television image to the television 2. For example, the CPU10 may output a Web page image in whole or in part to the television 2.Concretely, a Web page displayed on the terminal device 7 includes avideo, the CPU 10 may successively output images included in the videoto the television 2. Thus, the user can view the video on the Web pageon the television 2 having a larger screen.

Note that in the present example embodiment, the browser image isdisplayed at least on the terminal device 7, but the browser image maybe displayed on either the terminal device 7 or the television 2, orboth. When the browser image is displayed on the terminal device 7, thetelevision 2 can be advantageously used for another purpose. Inaddition, when the browser image is displayed on the television 2, thebrowser image can be readily viewed due to a large screen of thetelevision 2, which, for example, facilitates easy viewing of a video ona Web page displayed on the television 2. Moreover, in another exampleembodiment, the user may select a display device to which the browserimage is outputted.

Following step S35, the process of step S32 is performed again.Thereafter, until the result of the determination of step S33 or S34turns out to be affirmative, the browser process is repeatedlyperformed.

In the processes (menu process, television process, and browser process)in the initial state described above, the game apparatus 3 can generatean app selection image, a device control image, and a browser image asmain operation images. The app selection image, the device controlimage, and the browser image are outputted to the terminal device 7 oneat a time while being switched therebetween in accordance with apredetermined change operation. Thus, by change operations, the user canreadily switch among any of the screen for selecting an application, thescreen for viewing a program guide and operating the television, and thescreen for browsing Web pages, so that the game system 1 can be used forvarious purposes. In the present example embodiment, while in any one ofthe three initial states, two icons for transitioning to the other twostates are always displayed, so that, with a single operation ofselecting an icon, the user can quickly cause the game system 1 totransition to a desired state. Moreover, in the present exampleembodiment, to view a television program by transitioning from the statefor application selection or Web page browsing by using the game system1, the user may simply perform a change operation to display a devicecontrol image on the terminal device 7 and perform a control instructionoperation, so that switching operations can be readily performed usingthe terminal device 7.

(Interruption-Time Process)

Next, the interruption-time process will be described in detail. FIG. 22is a flowchart illustrating a flow of the app interruption processperformed by the game apparatus 3. In the app execution state, the appexecution process is performed with its processing details according toan application. Here, when a predetermined interruption operation isperformed in the app execution state, the CPU 10 interrupts (suspends)the app execution process, and performs the app interruption processshown in FIG. 22. The predetermined interruption operation may be anyoperation on the terminal device 7, but in the present exampleembodiment, it is an operation of pressing a home button 54C. Note thatthe home button 54C may be set to be not used for operating theapplication, so that the user can cause a transition to the interruptedstate at any time during the execution of the application.

Note that in the present example embodiment, the execution of theapplication itself is interrupted (suspended) in the interrupted state.However, in another example embodiment, the execution of the applicationmay be continued. For example, the CPU 10 may execute informationprocessing for the application to such an extent as not to affect use bythe user, including the execution of calculation for internal processingof the application.

In the app interruption process, the CPU 10 initially in step S41displays a setting image on the terminal device 7. Specifically, the CPU10 generates a setting image, and outputs (transmits) the generatedsetting image to the terminal device 7. Accordingly, in the presentexample embodiment, the first interrupted state to which the game system1 transitions is the setting state. Note that in another exampleembodiment, the first interrupted state to which the game system 1transitions may be the second television operation state. Moreover, thesetting image may be any operation image representing a settingoperation on the game system 1 (the terminal device 7). Hereinafter,referring to FIG. 23, the setting image in the present exampleembodiment will be described.

FIG. 23 is a diagram illustrating an example setting image displayed onthe terminal device 7. As shown in FIG. 23, the setting image includesan app end button 151, an app suspension button 152, an app resumptionbutton 153, a remote control display button 154, a terminal settingbutton 155, a controller setting button 156, a list button 157, and amanual button 158.

The app end button 151 is an image representing an instruction toterminate an application whose execution is interrupted. The appsuspension button 152 is an image representing an instruction to suspendthe application. The app resumption button 153 is an image representingan instruction to resume the application. With these buttons 151 to 153,it is possible to terminate or suspend the application, thereby causinga transition from the interrupted state to the menu state, or resume theapplication from the interrupted state.

The remote control display button 154 is an image representing aninstruction to display a remote control image (FIG. 24) on the terminaldevice 7. As will be described in detail later, the remote control imageis an operation image for operating the television 2, which includes,for example, button images representing operations on the television 2.

The terminal setting button 155 is an image representing an instructionto display a screen with which to make various settings for the terminaldevice 7. The controller setting button 156 is an image representing aninstruction to display a screen with which to make various settings forthe controller 5. The list button 157 is an image representing aninstruction to display a list of other registered users (acquaintancesand friends). For example, the user can play a game against orcommunicate with another user selected from the list. The manual button158 is an image representing an instruction to display a manual of thegame system 1 (or the terminal device 7). With these buttons 155 to 158,the user changes various settings for the game system 1.

In step S41, the CPU 10 and the GPU 11 b collaborate to read data foruse in generating a setting image, including, for example, images of thebuttons 151 to 158, from the VRAM 11 d, and generate the setting imagesusing the data being read. Thereafter, the generated setting images arestored to the VRAM 11 d. Moreover, the CPU 10 outputs the setting imagesto the terminal device 7. The process of outputting and displaying animage on the terminal device 7 is the same as in step S12. As a result,the setting image is displayed on the LCD 51. Moreover, in step S41,sound data may be transmitted to the terminal device 7 along with theimage data, so that the speakers 77 of the terminal device 7 outputsound. Following step S41, the process of step S42 is performed.

In step S42, the CPU 10 acquires various data from the terminal device7. The process of step S42 is the same as the process of step S11.Following step S42, the process of step S43 is performed.

In step S43, the CPU 10 determines whether or not to display a remotecontrol image. Specifically, the CPU 10 determines whether or not anyoperation has been performed to cause a transition from the settingstate to the second television operation state. This operation may beperformed in any arbitrary manner, and in the present exampleembodiment, it is an operation of touching the remote control displaybutton 154 displayed on the terminal device 7. Specifically, the CPU 10reads the touch data in the terminal operation data 104 from the mainmemory, and determines whether or not the remote control display button154 displayed on the terminal device 7 has been touched, on the basis ofthe touch data. When the result of the determination of step S43 isaffirmative, the process of step S44 is performed. On the other hand,when the result of the determination of step S43 is negative, theprocess of step S44 is skipped, and the process of step S45 isperformed.

In step S44, the CPU 10 displays a remote control image on the terminaldevice 7. Specifically, the CPU 10 generates a remote control image, andoutputs (transmits) the generated remote control image to the terminaldevice 7. The remote control image may be any operation imagerepresenting operations on the television 2. Hereinafter, referring toFIG. 24, the remote control image in the present example embodiment willbe described.

FIG. 24 is a diagram illustrating an example remote control imagedisplayed on the terminal device 7. As shown in FIG. 24, the remotecontrol image includes the app end button 151 and the app suspensionbutton 152 as included in the setting image. Accordingly, in the secondtelevision operation state of the present example embodiment, as in thesetting state, it is possible to terminate or suspend an application andtransition to the menu state. Note that in another example embodiment,for example, the CPU 10 may cause the terminal device 7 to display theapp resumption button 153 as displayed in the setting image, therebymaking it possible to resume an application and transition from thesecond television operation state to the app execution state.

Furthermore, the back button 167 is an image representing an instructionto transition (return) from the second television operation state to thesetting state. Specifically, in the present example embodiment, the usercan alternatingly display the setting image and the remote control imageby operations of selecting the remote control display button 154 or theback button 167.

Furthermore, the remote control image includes a power button 161, aninput switch button 162, a volume increment button 163, a volumedecrement button 164, a channel forward button 165, and a channel backbutton 166. The buttons 161 to 166 are images representing instructionsto control the television 2. The power button 161 is an imagerepresenting an instruction to switch on/off the television 2. The inputswitch button 162 is an image representing an instruction to switchbetween inputs to the television 2 (i.e., switch between a mode in whichtelevision program video is inputted and displayed and a mode in whichimages from the game apparatus 3 are inputted and displayed). The volumeincrement button 163 is an image representing an instruction to turn upthe volume of the television 2. The volume decrement button 164 is animage representing an instruction to turn down the volume of thetelevision 2. The channel forward button 165 is an image representing aninstruction to change the channel of the television 2 one by one inascending order. The channel back button 166 is an image representing aninstruction to change the channel of the television 2 one by one indescending order. With the buttons 161 to 166, the user can operate thetelevision 2 in terms of power on/off, input switching, volume, andchannel selection. Note that in another example embodiment, in additionto the aforementioned buttons, the terminal device 7 may display buttonsas provided on general television remote controls, including, forexample, buttons representing channels, a button representing aninstruction to display a program guide acquired from televisionbroadcasting, and a button representing a recording instruction (whenthe television 2 has the recording function).

As described above, in the present example embodiment, the remotecontrol image is an image including images representing various controlinstructions. Note that in another example embodiment, the remotecontrol image may be an image representing the content of controlinstructions (e.g., volume increment), and assignment of the controlinstructions to buttons (e.g., the second R button 54L). Moreover, theremote control image may include a program guide, as in the devicecontrol image.

In step S44, the CPU 10 and the GPU 11 b collaborate to read data foruse in generating a remote control image, including, for example, imagesof the buttons 151 to 153 and 161 to 166, from the VRAM 11 d, andgenerate a remote control image using the data being read. Thereafter,the generated remote control image is stored to the VRAM 11 d. Moreover,the CPU 10 outputs the remote control image to the terminal device 7.The process of outputting and displaying an image on the terminal device7 is the same as in step S12. As a result, the remote control image isdisplayed on the LCD 51. The process of step S44 causes the game system1 to transition from the setting state to the second televisionoperation state. Moreover, in step S44, sound data may be transmitted tothe terminal device 7 along with the image data, so that the speakers 77of the terminal device 7 output sound. Following step S44, the processof step S45 is performed.

Note that in another example embodiment, upon transition to the secondtelevision operation state, inputs to the television 2 may be switched(so that a television broadcast is displayed). Specifically, in the casewhere the television 2 can be switched between a first mode in whichimages inputted from the game apparatus 3 are displayed and a secondmode in which television broadcast images are displayed, the CPU 10 maycontrol the television 2 to switch to the second mode in response to theterminal device 7 displaying a sub-operation image (more specifically,the remote control image). Concretely, in step S44, the CPU 10 maycontrol input switching of the television 2 so that the television 2 isset to the second mode in which a television broadcast is displayed. Asa result, upon transition to the second television operation state, theuser does not perform any operation to switch between inputs to thetelevision 2, resulting in more simplified user operations. Note thatinput switching control may be performed by either the first or secondmethod described in conjunction with step S24.

In step S45, the CPU 10 determines whether or not any controlinstruction has been provided. The control instruction is an instructionto control the television 2. The operation to provide the controlinstruction may be arbitrary. In the present example embodiment, as instep S23, the control instruction is provided by an operation oftouching any one of the buttons 161 to 166 displayed on the terminaldevice 7 and representing control instructions. Specifically, the CPU 10reads the touch data in the terminal operation data 104 from the mainmemory, and determines whether or not any one of the buttons 161 to 166displayed on the terminal device 7 has been touched, on the basis of thetouch data. When the result of the determination of step S45 isaffirmative, the process of step S46 is performed. On the other hand,when the result of the determination of step S45 is negative, theprocess of step S46 is skipped, and the process of step S47 isperformed.

In step S46, the CPU 10 controls the television 2 in accordance with thecontrol instruction. The process of step S46 is the same as the processof step S24. Specifically, the CPU 10 controls the television 2 by thefirst method in which the game apparatus 3 outputs a control signal tothe television 2 via the terminal device 7 and/or the second method inwhich the game apparatus 3 outputs a control signal directly to thetelevision 2. Following step S46, the process of step S47 is performed.

In step S47, the CPU 10 determines whether or not to display a settingimage. Specifically, the CPU 10 determines whether or not any operationhas been performed to transition from the second television operationstate to the setting state. This operation may be performed in anyarbitrary manner, and in the present example embodiment, it is anoperation of touching the back button 167 displayed on the terminaldevice 7. Specifically, the CPU 10 reads the touch data in the terminaloperation data 104 from the main memory, and determines whether or notthe back button 167 displayed on the terminal device 7 has been touched,on the basis of the touch data. When the result of the determination ofstep S47 is affirmative, the process of step S48 is performed. On theother hand, when the result of the determination of step S47 isnegative, the process of step S48 is skipped, and the process of stepS49 is performed.

In step S48, the CPU 10 displays the setting image on the terminaldevice 7. The process of step S48 is the same as the process of stepS41. The process of step S48 causes the game system 1 to transition fromthe second television operation state to the setting state. Followingstep S48, the process of step S49 is performed.

In step S49, the CPU 10 performs various setting processing.Specifically, the CPU 10 determines whether or not the terminal settingbutton 155, the controller setting button 156, the list button 157, orthe manual button 158 has been selected (touched). If any one of thebuttons has been selected, processing corresponding to the selectedbutton is executed. The setting processing in step S49 may be similar tosetting processing as executed in conventional and general game systems.Following step S49, the process of step S50 is performed.

In step S50, the CPU 10 determines whether or not to transition to themenu state. Specifically, it is determined whether or not to terminateor suspend the application whose execution is interrupted. Thisdetermination is as to whether or not any operation has been performedto terminate or suspend the application (i.e., the operation causing atransition from the interrupted state to the menu state). This operationmay be performed in any arbitrary manner, and in the present exampleembodiment, it is an operation of touching the app end button 151 or theapp suspension button 152 displayed on the terminal device 7.Specifically, the CPU 10 reads the touch data in the terminal operationdata 104 from the main memory, and determines whether or not the app endbutton 151 or the app suspension button 152 displayed on the terminaldevice 7 has been touched, on the basis of the touch data. When theresult of the determination of step S50 is affirmative, the process ofstep S51 is performed. On the other hand, when the result of thedetermination of step S50 is negative, the process of step S52 isperformed.

In step S51, the CPU 10 performs an application closing process.Concretely, application data to be saved is stored to the main memory assave data 111. Note that the save data 111 is stored to the flash memory17 or a storage medium having the application stored therein, at anappropriate time. Moreover, when the app end button 151 is determined tohave been selected in step S50, the application starts from apredetermined initial state at the next startup. On the other hand, whenthe app suspension button 152 is determined to have been selected instep S50, the application starts from the interruption-time state at thenext startup (or at the resumption). Accordingly, the CPU 10 may havedifferent data stored as save data 111 between the case where the append button 151 is selected and the case where the app suspension button152 is selected. Following step S51, the CPU 10 ends theinterruption-time process. Thereafter, the game system 1 transitionsfrom the interrupted state to the menu state. That is, the CPU 10performs the menu process (FIG. 16).

As described above, in the present example embodiment, when apredetermined operation (interruption operation) is performed on theterminal device 7 during the execution of an application, the CPU 10generates a sub-operation image including a termination instructionimage (app end button 151) which represents an instruction to terminatethe application being executed (step S41). Thereafter, when aninstruction is provided to terminate the application by selecting theterminate instruction image, the execution of the application isterminated (step S50). Furthermore, when the execution of theapplication is terminated, the CPU 10 generates a main operation image.In this manner, in the present example embodiment, when the applicationbeing executed is terminated, the terminal device 7 displays a mainoperation image (more specifically, an app selection image). Thus, afterterminating an application, the user can readily select the nextapplication to be executed.

Note that, in the present example embodiment, when an application isterminated in the app interruption process, the game system 1transitions to the menu state. In another example embodiment, the gamesystem 1 may transition to the first television operation state or thebrowser state in the same situation. For example, in response to apredetermined operation being performed in the interrupted state, thegame system 1 may be able to transition to the first televisionoperation state or the browser state. Concretely, the CPU 10 may displaythe television icon 125 and/or the browser icon 126 both in the settingimage and the remote control image. Thereafter, when an operation isperformed to select the television icon 125, the CPU 10 terminates theapplication and displays a device control image on the terminal device7. As a result, the game system 1 transitions to the first televisionoperation state. Alternatively, when an operation is performed to selectthe browser icon 126, the CPU 10 terminates the application and displaysa browser image on the terminal device 7. As a result, the game system 1transitions to the browser state. In this manner, the user can allow thegame system 1 to transition from the interrupted state to a desired oneof the three initial states. In addition, for example, when theapplication is terminated, the CPU 10 may determine one of the threestates to which a transition is to be made, in accordance with apredetermined condition. Concretely, the CPU 10 may set the state towhich a transition is to be made as the initial state.

In step S52, the CPU 10 determines whether or not to resume theapplication whose execution is interrupted. Specifically, the CPU 10determines whether or not any operation has been performed to resume theapplication (i.e., to cause a transition from the interrupted state tothe app execution state). This operation may be performed in anyarbitrary manner, and in the present example embodiment, it is anoperation of touching the app resumption button 153 displayed on theterminal device 7. Specifically, the CPU 10 reads the touch data in theterminal operation data 104 from the main memory, and determines whetheror not the app resumption button 153 displayed on the terminal device 7has been touched, on the basis of the touch data. When the result of thedetermination of step S52 is affirmative, the CPU 10 ends theinterruption-time process, and performs the app execution process. As aresult, the execution of the application is resumed, and the game system1 transitions from the interrupted state to the app execution state. Onthe other hand, when the result of the determination of step S52 isnegative, the process of step S42 is performed again. Thereafter, untilthe result of the determination of step S50 or S52 turns out to beaffirmative, the interruption-time process is repeatedly performed.

In another example embodiment, upon transition from the secondtelevision operation state to the app execution state, inputs to thetelevision 2 may be switched (such that images from the game apparatus 3are displayed). Specifically, when the television 2 can be switchedbetween the first mode in which images inputted from the game apparatus3 are displayed and the second mode in which television broadcast imagesare displayed, the CPU 10 may control the television 2 to be switched tothe first mode, in response to the image displayed on the terminaldevice 7 being switched from the remote control image to an imagegenerated by executing an application. Concretely, when the result ofthe determination of step S52 is affirmative, the CPU 10 may performinput switching control such that the television 2 is brought into thefirst mode to display images from the game apparatus 3. As a result,upon transition from the second television operation state to the appexecution state, the user does not perform any operation to switchbetween inputs to the television 2, resulting in more simplified useroperations. Moreover, in another example embodiment, upon transitionfrom the second television operation state to any of the setting imagestate, the menu state, and the browser state, the CPU 10 may perform thecontrol as mentioned above. Note that the input switching control may beperformed by either the first or second method described in conjunctionwith step S24.

As described above, in the present example embodiment, when apredetermined operation (interruption operation) is performed on theterminal device 7 during the execution of an application, the CPU 10suspends the execution of the application. Moreover, in the state(interrupted state) where the terminal device 7 displays a sub-operationimage, the CPU 10 determines whether or not any resumption instructionhas been provided to resume the execution of the application (i.e., theoperation of selecting the app resumption button 153 has beenperformed), on the basis of the terminal operation data. When there hasbeen any resumption instruction, the CPU 10 resumes the execution of theapplication. In this manner, in the present example embodiment, when theexecution of the application is interrupted, the user can readily resumethe application by a resumption instruction.

As described above, in the present example embodiment, when apredetermined operation (interruption operation) is performed on theterminal device 7 during the execution of an application, the CPU 10generates a predetermined sub-operation image. Thereafter, in the state(interrupted state) where the terminal device 7 displays thesub-operation image on the display section, it is determined whether ornot any control instruction has been provided, on the basis of theterminal operation data (step S45). Accordingly, in the present exampleembodiment, when utilizing an application on the game system 1 using theterminal device 7, the user can operate the television 2 by interruptingthe usage of the application. As a result, the user can operate thetelevision 2 without ending the execution of the application, resultingin easier operations of switching between the state where the television2 is used for the game system 1 and the state where the television 2 isused to display a television program. For example, using the terminaldevice 7 makes it easy for the user to temporarily interrupt the usageof the application and switch between inputs to the television 2 to viewa television program, and also to temporarily interrupt televisionprogram viewing to use the application.

Furthermore, in the present example embodiment, the remote control imageis also displayed in the browser state. Specifically, in the processingloop of steps S32 to S35 in the browser process (FIG. 20), the CPU 10determines whether or not any operation has been performed to displaythe remote control image, concretely, the CPU 10 determines whether ornot the operation of selecting (touching) the remote control displaybutton 147 has been performed. When the result of the determination isaffirmative, the same process as in step S44 is performed. Consequently,the terminal device 7 displays the remote control image. Note that theremote control image displayed in the browser state includes neither theapp end button 151 nor the app suspension button 152. When the backbutton 167 is selected, the CPU 10 displays the browser image on theterminal device 7. In this manner, in the present example embodiment,the user can display the remote control image and operate the television2 in the browser state as well.

Furthermore, in another example embodiment, a transition can be madefrom any of the three initial states to the state where sub-operationimages (the setting image and the remote control image) can bedisplayed. Specifically, when a predetermined operation is performed inany of the three states, the CPU 10 may perform a process similar to theinterruption-time process. Note that in this case, the app end button151 and the app suspension button 152 are included neither in thesetting image nor in the remote control image. Moreover, a button imagerepresenting an instruction to transition back to the original state isdisplayed in place of the app resumption button 153. Thus, thesub-operation image can be readily displayed in the initial state aswell, so that the user can more readily operate the television 2 in theinitial state.

Note that in another example embodiment, in addition to the remotecontrol image being displayed on the terminal device 7, a program guideimage may be displayed on the television 2 in the interrupted state.Specifically, when the terminal device 7 displays a sub-operation image,the CPU 10 may generate and output an image including a televisionprogram guide to the television 2. Note that the program guide image maybe acquired from the external device 91 via the network 90, as in thetelevision process. Moreover, the program guide may be displayed on thetelevision 2 at any arbitrary time. For example, the program guide maybe displayed on the television 2 at the start of the interruption-timeprocess, at the time of the remote control image being displayed on theterminal device 7, or at the time of a predetermined display instructionbeing provided by the user.

(Operations to Shut Down the Game System 1)

In the present example embodiment, the user can shut down the gamesystem 1 at any arbitrary time during the information processing (theinitial process, the menu process, the television process, the browserprocess, the app execution process, and the interruption-time process).Specifically, when a predetermined shutdown operation is performed(e.g., the power button 54M of the terminal device 7 or the power button24 of the game apparatus 3 is pressed), the CPU 10 performs apredetermined shutdown process, so that the game apparatus 3 and theterminal device 7 are powered off. Note that the shutdown processincludes processing of storing data representing the state of the gamesystem 1 (e.g., the menu state) at the time of shutdown to the flashmemory 17. On the basis of that data, the initial state can bedetermined at the next boot.

With the information processing as described above, the user can use thegame system 1 for various purposes, including, for example, programguide browsing, Web page browsing, and use of various applications.Moreover, by using the screen of the terminal device 7 as an operationscreen, television 2 can be used for other purposes, including, forexample, viewing a television program while using the game system 1, andtherefore, the game system 1 and the television 2 can be used at thesame time. As a result, the user can smoothly switch between the use ofthe game system 1 and the use of the television 2 (for a purpose otherthan in the game system 1). For example, the user can more smoothlyenjoy using the game system 1 while watching a television program, or inturn can more smoothly enjoy watching a television program while usingthe game system 1. Moreover, both the game system 1 and the television 2can be operated by the terminal device 7, which makes it possible toreadily perform an operation to switch between the state where the gamesystem 1 is used and the state where the television is used (for apurpose other than in the game system 1).

7. Variants

The above example embodiment is merely illustrative, and in anotherexample embodiment, for example, the game system or suchlike can becarried out with, for example, a configuration as described below.

(Variant Related to the Initial State)

In the above example embodiment, the initial state includes three stateswhich are different in terms of images to be displayed on the terminaldevice 7. Here, in another example embodiment, the number of stateincluded in the initial states is not limited. Moreover, these statesmay or may not be different in terms of images to be displayed on theterminal device 7. For example, the states may be the same in terms ofthe images to be displayed on the terminal device 7 but may be differentin terms of operations that can be performed.

(Variant Related to the Predetermined Display Device Used in the GameSystem 1)

In the above example embodiment, the predetermined display device usedin the game system 1 has been described taking as an example thetelevision 2 capable of receiving television broadcasting and displayinga television program. Here, the predetermined display device is notlimited to the television 2, and may be any device having a plurality ofinput terminals and capable of image display. For example, thepredetermined display device may be a display device used as a monitorof a personal computer (different from the game apparatus 3) or amonitor of a recorder/player.

(Variant Related to the Configuration of the Game System 1)

In the above example embodiment, the game system 1 is configured toinclude the game apparatus 3, the terminal device 7 provided with thedisplay section (LCD 51), the controller 5, and the television 2. Here,the game system may be configured to include a game apparatus and anoperating device, and may be capable of displaying images on a displaysection of the operating device and a predetermined display device. Forexample, the game system may be provided in a form not including thecontroller 5 or the predetermined display device (television 2).

Furthermore, the example embodiment has been described taking as anexample the game system including a game apparatus capable of executinga game program, but another example embodiment may be an arbitraryinformation processing system and apparatus for use in executing anyarbitrary application. Specifically, in the above example embodiment,the one or more applications executable by the game apparatus include agame application, but it is possible that no game application isincluded.

(Variant Related to the Information Processing Apparatus for Executingthe Game Processing)

In the above example embodiment, a series of information processingtasks to be performed in the game system 1 is executed by the gameapparatus 3, but the series of information processing tasks may beexecuted in part by another device. For example, in another exampleembodiment, a part (e.g., the terminal image generation process or theapplication execution process) of the series of information processingtasks may be executed by the terminal device 7. Moreover, in anotherexample embodiment, a series of information processing tasks in an inputsystem including a plurality of information processing apparatusescapable of communicating with each other may be shared between theinformation processing apparatuses. Specifically, the game systemincludes a game apparatus and an operating device, and may also include(a) an acquisition section for acquiring operation data representing anoperation on the operating device, (b) a first operation imagegeneration section for generating a predetermined first operation imageto be displayed on a display unit in an initial state after boot of thegame apparatus, (c) an operation image output section for outputting thefirst operation image to the operating device, (d) an instructiondetermination section for making determinations in the initial state onthe basis of the operation data, the determinations being as to whetheror not a selection instruction has been provided to select anapplication from among one or more applications, including a gameapplication, which are executable by the game apparatus, and whether ornot a control instruction has been provided to control a predetermineddisplay device different from the operating device, (e) an applicationexecution section for, when the selection instruction is determined tohave been provided, executing the selected application, and outputtingan image generated by the execution of the application to either theoperating device or the predetermined display device, or both, and (f) adevice control section for, when the control instruction is determinedto have been provided, controlling the predetermined display device inaccordance with the control instruction. In this case, one or moredevices may include the sections (a) to (f).

Note that in the case where the aforementioned information processing isshared between information processing apparatuses, when processing issynchronized between the information processing apparatuses, theinformation processing becomes complicated. On the other hand, in thecase where, as in the above example embodiment, information processingis executed by one game apparatus 3, and the terminal device 7 performsprocesses of receiving and displaying images (i.e., the terminal device7 functions as a thin-client terminal), the information processing isnot synchronized between information processing apparatuses, resultingin simplified information processing.

The systems, devices and apparatuses described herein may include one ormore processors, which may be located in one place or distributed in avariety of places communicating via one or more networks. Suchprocessor(s) can, for example, use conventional 3D graphicstransformations, virtual camera and other techniques to provideappropriate images for display. By way of example and withoutlimitation, the processors can be any of: a processor that is part of oris a separate component co-located with the stationary display and whichcommunicates remotely (e.g., wirelessly) with the movable display; or aprocessor that is part of or is a separate component co-located with themovable display and communicates remotely (e.g., wirelessly) with thestationary display or associated equipment; or a distributed processingarrangement some of which is contained within the movable displayhousing and some of which is co-located with the stationary display, thedistributed portions communicating together via a connection such as awireless or wired network; or a processor(s) located remotely (e.g., inthe cloud) from both the stationary and movable displays andcommunicating with each of them via one or more network connections; orany combination or variation of the above.

The processors can be implemented using one or more general-purposeprocessors, one or more specialized graphics processors, or combinationsof these. These may be supplemented by specifically-designed ASICs(application specific integrated circuits) and/or logic circuitry. Inthe case of a distributed processor architecture or arrangement,appropriate data exchange and transmission protocols are used to providelow latency and maintain interactivity, as will be understood by thoseskilled in the art.

Similarly, program instructions, data and other information forimplementing the systems and methods described herein may be stored inone or more on-board and/or removable memory devices. Multiple memorydevices may be part of the same device or different devices, which areco-located or remotely located with respect to each other.

As described above, the above example embodiment can be applied to, forexample, a game system, a game apparatus, and an information processingprogram for the purpose of, for example, allowing a display device, suchas a television, to be readily operated during the execution of anapplication in the game system.

While certain example systems, methods, devices and apparatuses havebeen described herein, it is to be understood that the appended claimsare not to be limited to the systems, methods, devices and apparatusesdisclosed, but on the contrary, are intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

What is claimed is:
 1. An information processing system comprising afirst information processing device and a second information processingdevice, wherein the first information processing device includes: afirst communication interface configured for communication with thesecond information processing device; and a first processing system,including at least one processor, for executing an application based onfirst operation data representing an operation performed on the firstinformation processing device and second operation data received fromthe second information processing device using the first communicationinterface, and displaying an image generated by execution of theapplication on a predetermined display device different from the secondinformation processing device, the second information processing deviceincludes: a second communication interface configured for communicationwith the first information processing device; a second processingsystem, including at least one processor, for executing an applicationbased at least on the second operation data representing an operationperformed on the second information processing device; and a display fordisplaying an image generated by execution of the application, and if apredetermined operation is performed on the second informationprocessing device while the application is executed on the secondinformation processing device, the second processing system generates anoperation image and displays the operation image on the display, and ifa control instruction is provided while the operation image is displayedon the display, the second processing system controls the predetermineddisplay device according to the control instruction.
 2. The informationprocessing system according to claim 1, wherein in a state where theoperation image is displayed on the display, the second processingsystem determines whether or not a termination instruction has beenprovided to terminate the application being executed, and when thetermination instruction is determined to have been provided, the secondprocessing system terminates the execution of the application.
 3. Theinformation processing system according to claim 1, wherein when apredetermined operation is performed on the second informationprocessing device during the execution of the application, the secondprocessing system suspends the execution of the application, in thestate where the operation image is displayed on the display, the secondprocessing system determines whether or not a resumption instruction hasbeen provided to resume the execution of the application, and when theresumption instruction is determined to have been provided, the secondprocessing system resumes the execution of the application.
 4. Theinformation processing system according to claim 1, wherein thepredetermined display device is configured to receive televisionbroadcasting and display a television program, and when the operationimage is displayed on the display, the first processing system generatesan image including a television program guide and outputs the generatedimage to the predetermined display device.
 5. The information processingsystem according to claim 1, wherein the second processing systemgenerates as the operation image an image including a controlinstruction image representing the control instruction, and when thecontrol instruction image is selected, the second processing systemdetermines the control instruction to have been provided.
 6. Theinformation processing system according to claim 1, wherein thepredetermined display device is configured to receive televisionbroadcasting and display a television program, and the second processingsystem generates an image including a television program guide as theoperation image, and outputs the generated image to the display.
 7. Theinformation processing system according to claim 1, wherein thepredetermined display device is configured to receive televisionbroadcasting and display a television program, the predetermined displaydevice is switchable between a first mode in which an image is inputtedfrom the first information processing device is displayed and a secondmode in which a television broadcast image is displayed, and the secondprocessing system controls the predetermined display device to beswitched to the second mode in response to the operation image beingdisplayed on the display.
 8. The information processing system accordingto claim 1, wherein the predetermined display device is configured toreceive television broadcasting and display a television program, thepredetermined display device is switchable between a first mode in whichan image inputted from the first information processing device isdisplayed and a second mode in which a television broadcast image isdisplayed, and the second processing system controls the predetermineddisplay device to be switched to the first mode in response to the imagedisplayed on the display switching from the operation image to an imagegenerated by execution of the application.
 9. The information processingsystem according to claim 1, wherein the second information processingdevice further includes a control signal transmitter for transmitting acontrol signal to control the predetermined display device, and thesecond processing system causes the control signal transmitter totransmit a control signal indicating the content of control according tothe control instruction.
 10. The information processing system accordingto claim 9, wherein the control signal transmitter transmits an infraredsignal as the control signal, and the predetermined display deviceincludes an infrared light receiver for receiving the infrared signal.11. The information processing system according to claim 1, wherein thesecond information processing device further includes a touch panelprovided on a screen of the display, the second processing systemgenerates as the operation image an image including a controlinstruction image representing the control instruction, and when thecontrol instruction image is selected on the touch panel, the secondprocessing system determines the control instruction to have beenprovided.
 12. The information processing system according to claim 1,wherein the control instruction is at least one of (a) an instruction totune to a channel, (b) an instruction to turn on or off a power supply,(c) an instruction to change a sound volume, and (d) an instruction tochange an input.
 13. A game system comprising a game apparatus and aterminal device, wherein the game apparatus includes: a firstcommunication interface configured for communication with the terminaldevice; and a first processing system, including at least one processor,for executing a game application based on first operation datarepresenting an operation performed on the game apparatus and secondoperation data received from the terminal device using the firstcommunication interface, and generating a game image by execution of theapplication, the terminal device includes: a second communicationinterface configured for communication with the game apparatus; a secondprocessing system, including at least one processor, for executing agame application based at least on second operation data representing anoperation performed on the terminal device; and a display for displayingthe game image generated by execution of the game application, whereinif a predetermined operation is performed on the terminal device whilethe game application is executed on the terminal device, the secondprocessing system generates an operation image and displays theoperation image on the display, and if a control instruction is providedwhile the operation image is displayed on the display, the secondprocessing system controls a predetermined display device different fromthe terminal device according to the control instruction.
 14. The gamesystem according to claim 13, wherein in a state where the operationimage is displayed on the display, the second processing systemdetermines whether or not a termination instruction has been provided toterminate the game application being executed, and when the terminationinstruction is determined to have been provided, the second processingsystem terminates the execution of the game application.
 15. The gamesystem according to claim 13, wherein when a predetermined operation isperformed on the terminal device during the execution of the gameapplication, the second processing system suspends the execution of thegame application, in the state where the operation image is displayed onthe display, the second processing system determines whether or not aresumption instruction has been provided to resume the execution of thegame application, and when the resumption instruction is determined tohave been provided, the second processing system resumes the executionof the game application.
 16. The game system according to claim 13,wherein the second processing system generates as the operation image animage including a control instruction image representing the controlinstruction, and when the control instruction image is selected, thesecond processing system determines the control instruction to have beenprovided.
 17. The game system according to claim 13, wherein thepredetermined display device is configured to receive televisionbroadcasting and display a television program, and the second processingsystem generates an image including a television program guide as theoperation image, and outputs the generated image to the terminal device.18. The game system according to claim 13, wherein the terminal devicefurther includes a control signal transmitter for transmitting a controlsignal to control the predetermined display device, and, the secondprocessing system causes the control signal transmitter to transmit acontrol signal indicating the content of control according to thecontrol instruction.
 19. The game system according to claim 18, whereinthe control signal transmitter transmits an infrared signal as thecontrol signal, and the predetermined display device includes aninfrared light receiver for receiving the infrared signal.
 20. The gamesystem according to claim 13, wherein the terminal device furtherincludes a touch panel provided on a screen of the display, the secondprocessing system generates as the operation image an image including acontrol instruction image representing the control instruction, when thecontrol instruction image is selected on the touch panel, the secondprocessing system determines the control instruction to have beenprovided.
 21. The game system according to claim 13, wherein the controlinstruction is at least one of (a) an instruction to tune to a channel,(b) an instruction to turn on or off a power supply, (c) an instructionto change a sound volume, and (d) an instruction to change an input. 22.An information processing device configured for communication withanother information processing device, wherein the other informationprocessing device executes an application based on first operation datarepresenting an operation performed on the other information processingdevice and second operation data received from the informationprocessing device, and displays an image generated by execution of theapplication on a predetermined display device different from theinformation processing device, the information processing deviceincludes: a communication interface configured for communication withthe other information processing device; a processing system, includingat least one processor, for executing an application based at least onsecond operation data representing an operation performed on theinformation processing device; and a display for displaying the imagegenerated by execution of the application, and if a predeterminedoperation is performed on the information processing device while theapplication is executed on the information processing device, theprocessing system generates an operation image and displays theoperation image on the display, and if a control instruction is providedwhile the operation image is displayed on the display, the processingsystem controls the predetermined display device according to thecontrol instruction.
 23. A non-transitory computer-readable storagemedium having stored therein an information processing program to beexecuted by a computer of an information processing device configuredfor communication with another information processing device, whereinthe other information processing device executes an application based onfirst operation data representing an operation performed on the otherinformation processing device and second operation data received fromthe information processing device, and displays an image generated byexecution of the application on a predetermined display device differentfrom the information processing device, the information processingprogram causes the computer to: transmit second operation datarepresenting an operation performed on the information processing deviceto the other information processing device using a communicationinterface; if a predetermined operation is performed on the informationprocessing device during the execution of the application based at leaston the second operation data representing an operation performed on theinformation processing device, generate an operation image and displaythe operation image on a display of the information processing device;and if a control instruction is provided while the operation image isdisplayed on the display, control the predetermined display deviceaccording to the control instruction.
 24. An information processingmethod for execution in an information processing system including afirst information processing device and a second information processingdevice, wherein the first information processing device: receives secondoperation data representing an operation performed on the secondinformation processing device using a communication interface; executesan application, on a processing system including at least one processor,based on first operation data representing an operation performed on thefirst information processing device and the received second operationdata; and displays an image generated by execution of the application ona predetermined display device different from the second informationprocessing device, the second information processing device: executes anapplication, on a processing system including at least one processor,based at least on second operation data representing an operationperformed on the second information processing device; displays an imagegenerated by execution of the application on a display of the secondinformation processing device; if a predetermined operation is performedon the second information processing device while the application isexecuted on the second information processing device, generates anoperation image and displays the operation image on the display; and ifa control instruction is provided while the operation image is displayedon the display, controls the predetermined display device according tothe control instruction.
 25. A terminal device configured forcommunication with a game apparatus, wherein the game apparatus executesa game application based on first operation data representing anoperation performed on the game apparatus and second operation datareceived from the terminal device using a communication interface, andgenerates a game image by execution of the game application, theterminal device includes: a second communication interface configuredfor communication with the game apparatus; a processing system,including at least one processor, for executing a game application basedat least on second operation data representing an operation performed onthe terminal device; and a display for displaying the game imagegenerated by execution of the game application, wherein if apredetermined operation is performed on the terminal device while thegame application is executed on the terminal device, the processingsystem generates an operation image and displays the operation image onthe display, and if a control instruction is provided while theoperation image is displayed on the display, the processing systemcontrols a predetermined display device different from the terminaldevice according to the control instruction.
 26. A non-transitorycomputer-readable storage medium having stored therein an informationprocessing program to be executed by a computer of a terminal deviceconfigured for communication with a game apparatus, wherein the gameapparatus executes a game application based on first operation datarepresenting an operation performed on the game apparatus and secondoperation data received from the terminal device using a communicationinterface, and generates a game image by execution of the gameapplication, the information processing program causes the computer to:transmit second operation data representing an operation performed onthe terminal device to the game apparatus using a communicationinterface; if a predetermined operation is performed on the terminaldevice during the execution of the game application on the terminaldevice based at least on the second operation data representing anoperation performed on the terminal device, generate an operation imageand display the operation image on a display of the terminal device; andif a control instruction is provided while the operation image isdisplayed on the display, control a predetermined display devicedifferent from the terminal device according to the control instruction.27. An information processing method for execution in a game systemincluding a game apparatus and a terminal device, wherein the gameapparatus: receives second operation data representing an operationperformed on the terminal device using a communication interface; andexecutes a game application, on a processing system including at leastone processor, based on first operation data representing an operationperformed on the game apparatus and the second operation data receivedfrom the terminal device, and generates a game image by execution of theapplication, the terminal device: executes a game application, on aprocessing system including at least one processor, based at least onsecond operation data representing an operation performed on theterminal device; displays a game image generated by execution of thegame application on a display of the terminal device; if a predeterminedoperation is performed on the terminal device while the game applicationis executed on the terminal device, generates an operation image anddisplays the operation image on the display; and if a controlinstruction is provided while the operation image is displayed on thedisplay, controls a predetermined display device different from theterminal device according to the control instruction.